Core technologies
SQL, NoSQL, MapReduce, Spark, and modern data platforms.
Course overview
From raw data to useful answers.
This course is an introduction to the principles, technologies, and challenges of big data management and analysis. We will work across the full data lifecycle: framing questions, collecting data, preparing it for analysis, choosing appropriate technologies, and presenting findings clearly.
Practical techniques
Data preprocessing, querying, benchmarking, visualization, and text analysis.
Real applications
Hands-on work grounded in real-world datasets, distributed computing, and machine learning.
Learning outcomes
- Compare core technologies for managing large-scale data.
- Choose and optimize data-management approaches for different scenarios.
- Extract insights, visualize results, and communicate a convincing data story.
Prerequisites
- Basic Python knowledge.
- Basic data-analysis skills, such as working with spreadsheets.
- A willingness to investigate messy, real-world data.
Class schedule
A high-level roadmap.
We meet every Thursday from 6:00 to 8:30 p.m. Topics and activities may evolve as the class progresses; readings, materials, and project details will be shared during the semester.
Attend in person or online
You may join class in person or online. No advance notice is needed. Use the class document to access the Zoom link.
Participation is essential
Ask thoughtful questions and join the discussion. After a good question or response, use the class document to access the participation form and record your contribution.
Class Slack group
Use the class Slack workspace to connect with classmates, ask questions, and share ideas. The invitation link is available in the class document.
Upcoming classes
October 1
Web data collection and database foundations Applying database fundamentals to web scraping, collection, and data management.
- Project submissions and September 24 class assignment review
- Wrap-up: NoSQL versus SQL comparison and trade-offs
- Web scraping, web data collection, and data quality
- Database basics in a web-data context
October 8
Machine learning and big data Machine learning and big data concepts and applications.
October 15
Data analysis and project development Exploration, validation, and refining data-driven questions.
October 22
Distributed computing foundations MapReduce, Spark, and scalable processing.
October 29
Spark in practice: building and optimizing distributed data workflows.
November 5
Text data and data challenges: unstructured data and large-scale analysis.
November 12
LLMs and urban analytics: guest perspectives and emerging applications.
November 19
AI/ML for big data: modern methods, evaluation, and responsible use.
November 26
No class: Thanksgiving recess.
December 3
Current big data trends Streaming, modern platforms, and project progress.
December 10
Final project presentations: findings, methods, and lessons learned.
Past classes
September 3
Introduction to big data Course foundations, questions, and data realities.
- Interactive format: live demos, Mentimeter, and real-time questions
- Big data beyond scale: people, documentation, policy, and privacy
- Data cleaning, labeling, provenance, and validation as central work
- Data cascades: early collection errors and bias amplified downstream
- Storage trade-offs: files, databases, single powerful machines, and distributed systems
- AI encouraged—with critical judgment and human oversight
- Course structure: participation, one open-resource quiz, data challenge, final project
September 10
End-to-end data analysis A practical walk-through from raw logs to usable data.
- Danny Smart: a simple, fictional e-commerce website
- Server-log extraction and inspection
- Building intuition about the data and its structure
- Field extraction and data cleaning
- Removing noisy and irrelevant log entries
- Debugging exercise: identifying a bug in the final code example
September 17
Yelp data analysis From project ideas and personal data to practical large-scale analysis.
- Follow-up debugging and interpretation of last week’s e-commerce data
- Final-project groups, original questions, and meaningful data-management challenges
- Using the class Slack group to discuss ideas and form project teams
- Extra-credit personal-data project: phone activity, health signals, and privacy trade-offs
- Yelp dataset exploration: JSON data, preprocessing, and hidden noise
- Tool trade-offs for analysis: Pandas, SQLite, and DuckDB
- Performance measurement and careful evaluation of AI-generated code
September 24
Yelp analysis and database trade-offs Comparing analysis tools, database systems, and performance strategies.
- AI context windows, human judgment, and asking good questions
- Slack for class questions, collaboration, and project-team formation
- Project ideation, novelty checks, iterative feedback, and concise visual storytelling
- Yelp analysis with Pandas, DuckDB, Spark, and SQLite
- Analytics engines versus transactional databases
- SQL schemas, indexes, query performance, and storage trade-offs
- Benchmarking indexed and non-indexed database queries
Assessment
Applied learning, assessed broadly.
Assessment emphasizes active participation, a contextual quiz, a data challenge, and a final project.
| Component | Points | How it is assessed |
|---|---|---|
| Participation, polls, and exercises | 20 points | Thoughtful questions and answers, class engagement, online polls, and after-class exercises. |
| In-class quizzes | 20 points | Two quizzes worth 10 points each: Quiz 1 on October 8 and Quiz 2 on December 3. |
| Big Data Challenge | 30 points | A timed data challenge emphasizing accuracy, efficiency, and scalable code. |
| Final project | 30 points | A collaborative, original data-driven project and final presentation. |
| Extra credit | 10 points | Opportunities and details will be announced during the semester. |
Grade cap and letter grades. These components total more than 100 points. Final scores will be capped at 100 points. Letter grades will follow NYU’s grading policy.
Quizzes: 20 points
There will be two 10-point quizzes: Quiz 1 on October 8 and Quiz 2 on December 3. Each quiz is completed synchronously through GradeScope, is open book and open internet, and lasts 30 minutes. Questions use practical scenarios discussed during the course and reward understanding the main ideas, not memorization.
Participation, polls, and exercises: 20 points
This category includes thoughtful questions and answers, active class engagement, occasional Mentimeter polls, and after-class exercises. After you answer a question or ask a good question in class, fill out the class document to access the Google Form and record your contribution. You may speak multiple times in a class, but participation is capped at one point per class. The goal is for every student to speak at least once in every class.
Big Data Challenge
Work individually or in a small group to solve data tasks. Submissions will be assessed for correctness and performance, with opportunities to refine and resubmit.
Final project
Develop an original, data-driven project. You will document your data pipeline, reflect on data quality and bias, and present a clear story with your findings. Use the class Slack workspace to discuss ideas and find or form a project group. The invitation link is in the class document.
Project milestones
- October 1: Submit a short preliminary idea and proposed team through a Google Form. The form link will be announced. Submit early so there is time for feedback and iteration; preliminary ideas will be discussed in class.
- October 15: Final deadline for preliminary project ideas and proposed teams. Submit through the Google Form; ideas will be discussed and finalized in class.
- October 22: Brief in-class presentation of each project plan, followed by feedback.
Support
Ask early. Build together.
Course communication
Use the class Slack workspace for questions, project discussion, and sharing useful resources. Access the Slack invitation through the class document.
Office hours
Danny Y. Huang will be available after class on Thursdays, 8:30–9:00 p.m., in person or on Zoom using the class link.