Generate Historical Stock Returns Summary with Airflow, Dask, and AWS S3

Table of Contents

1. Introduction
2. Airflow and Dash Overview
3. Generating Summary Statistics using Airflow and Dash
4. Understanding the Code Structure
5. Running the Airflow DAG
6. Analyzing the Summary Statistics
7. Uploading the Data to AWS S3
8. Exploring the Parquet File in AWS S3
9. Future Enhancements
10. Conclusion

Introduction

In today's session, we will discuss airflow, dash, and AWS S3 parquet file use cases. We will explore how to generate summary statistics for individual tickers using airflow and dash. The generated statistics will be saved as a parquet file in AWS S3. We will also cover the code structure, running the airflow DAG, analyzing the summary statistics, uploading the data to AWS S3, and exploring the parquet file in the AWS S3 console.

Airflow and Dash Overview

Airflow is a platform used to programmatically author, schedule, and monitor workflows. It allows you to orchestrate complex tasks and data pipelines in a scalable manner. Dash, on the other hand, is a productive framework for building analytical web applications. It provides a convenient interface for creating interactive dashboards and visualizations.

Generating Summary Statistics using Airflow and Dash

To generate summary statistics for individual tickers, we utilize airflow and dash. We start by obtaining a list of tickers from various sources, such as S&P 500, NASDAQ 100, Russell 2000, Russell 3000, and Wilshire 5000. These tickers are then processed in parallel using multiple dash worker nodes. Each worker node calculates the summary statistics for an individual ticker and pushes the data to AWS S3 or other compatible storage service.

Understanding the Code Structure

The code for generating summary statistics can be found in the summary_stats.py file. It consists of a main function called get_summary_stats that takes a list of tickers as input. The function utilizes several libraries, including PCA engine and pi folio, to retrieve and process the performance statistics of individual stocks. The resulting data frames are then converted into a dictionary and uploaded to AWS S3 or countable. The column names of the data frames are manipulated to ensure compatibility with JSON format.

Running the Airflow DAG

To execute the airflow DAG, we trigger it manually using the airflow UI. The DAG is scheduled to run every night during weekdays and on Saturdays. Each tickers' summary statistics are processed in parallel, resulting in a comprehensive data frame containing the combined statistics. The final data frame is then uploaded as a parquet file to AWS S3.

Analyzing the Summary Statistics

The generated summary statistics provide valuable insights into the performance of individual tickers. Key metrics such as annual return, cumulative return, annual volatility, and various ratios are included. These statistics can be used for automated trading strategies and decision-making in the financial domain.

Uploading the Data to AWS S3

Once the summary statistics are calculated, they are pushed to AWS S3 using the upload_stats_to_aws function. The parquet file is stored in the specified S3 path for future reference and analysis. AWS Data Wrangler is used to convert the data frame to a parquet file, simplifying the conversion process.

Exploring the Parquet File in AWS S3

The parquet file generated from the summary statistics is stored in the AWS S3 console. The file size is approximately 1MB for 5000 tickers. It provides a convenient and efficient way to store and analyze large datasets. In the future, additional tickers like ETFs, futures, and mutual funds can be included to enrich the dataset.

Future Enhancements

In the future, we plan to expand the list of tickers to include more financial instruments. This will provide a broader scope for analysis and decision-making. We will also explore the possibility of including additional features such as ETFs, futures, and mutual funds to create a more comprehensive dataset for analysis.

Conclusion

In conclusion, the integration of airflow, dash, and AWS S3 allows us to efficiently generate summary statistics for individual tickers. The use of parallel processing and distributed computing enables faster and more scalable data analysis. The resulting parquet file provides a convenient format for storing and analyzing large datasets. By leveraging these technologies, we can gain valuable insights into financial markets and make informed investment decisions.