{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.10.13","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"none","dataSources":[{"sourceId":56537,"databundleVersionId":8015876,"sourceType":"competition"}],"dockerImageVersionId":30732,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Hello, world!\n\nLEAP - Atmospheric Physics using AI\n\nHello!\n\nI am Emilly, a Junior Data Engineer\n\nI love Data Science, and I love any topic that involves space, mathematics, physics, chemistry and computing.\n\nWhen I was young I loved to play Space Station 13, and that's the main reason I loved the idea of this challenge, too!\n\nIn the game, you could choose your profession before every round, and Atmospherics Engineer was one of the most complicated jobs you could choose in the entire crew. (Aside from being the Captain)\n\n# Introduction\n\n## CSV Strategies\n\nThe main objective of the challenge is to create an AI algorithm that will predict target values of a **Data Science Problem**.\n\n> \"Your goal is to create a model that predicts the target variables associated with a given set of input variables.\" - Data page\n\nHowever, above that citation, there is a primary concern:\n\n> \"However, this multi-scale framework comes at a great computational cost, limiting its usage for experiments and ensemble climate projections. The goal is to train a model to emulate the effects of these small-scale processes at a fraction of the cost of explicitly resolving them.\" - Data page\n\nSo, if we want to improve the algorithm, we need to understand what are the fastest and most cost-efficient tools to deal with large-scale data\n\nThat is why I developed CSV Strategies\n\nThe challenge data is provided as CSV, and there is also an available API for downloading more data from HuggingFace, which is not implemented here yet\n\nHowever, this notebook will give you a good understanding of what tools to use when dealing with huge datasets\n\nBeing a Jr. Data Engineer I am very concerned about performance and I am very grateful to have found this challenge and the people who are submitting to it!\n\nHave a good read.\n\n![alt text](https://github.com/EmaoqFilho-NTTData/csv_strategy/blob/main/strategies_v5.png?raw=true \"Title\")","metadata":{}},{"cell_type":"code","source":"!pip install pyspark > NULL\nprint(\"installed pyspark\")\n!pip install dask > NULL\nprint(\"installed dask\")\n!pip install datatable > NULL\nprint(\"installed datatable\")\n!pip install ray > NULL\nprint(\"installed ray\")\n!pip install unidist > NULL\nprint(\"installed unidist\")\n!pip install modin > NULL\nprint(\"installed modin\")\n\n!pip install ipywidgets > NULL\nprint(\"installed ipywidgets\")\n\n# Library for abstraction\nfrom abc import ABC, abstractmethod\nprint(\"abstraction imported\")\n\n# Libraries for strategy choosing\nimport pandas as pd\nfrom pyspark.sql import SparkSession\nfrom io import StringIO\nimport polars as pl\nimport dask.dataframe as dd\nimport datatable as dtt\nimport modin.pandas as modpd\nimport os\nprint(\"strategies imported\")\n\n# util\nfrom datetime import datetime\nimport ipywidgets\nimport numpy as np\nprint(\"utils imported\")\n\n# create files dictionary\nfiles = {}\n\nprint(\"creating files {dictionary:values}...\")\n\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        file_path = os.path.join(dirname, filename)\n        print(file_path)\n        files[filename] = file_path","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2024-06-09T15:13:03.513523Z","iopub.execute_input":"2024-06-09T15:13:03.513987Z","iopub.status.idle":"2024-06-09T15:19:47.9963Z","shell.execute_reply.started":"2024-06-09T15:13:03.513953Z","shell.execute_reply":"2024-06-09T15:19:47.995301Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# utils\ndef now():\n    return datetime.now()","metadata":{"execution":{"iopub.status.busy":"2024-06-09T15:34:14.731903Z","iopub.execute_input":"2024-06-09T15:34:14.732682Z","iopub.status.idle":"2024-06-09T15:34:14.737664Z","shell.execute_reply.started":"2024-06-09T15:34:14.732641Z","shell.execute_reply":"2024-06-09T15:34:14.736533Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"###################################################\n#          CSV READ STRATEGY                      #\n###################################################\nclass CSVReadStrategy(ABC):\n    @abstractmethod\n    def read_lines(self, file_path,n_lines):\n        \"\"\" read a file's n_lines: number of lines\"\"\"\n        pass\n\n###################################################\n    \nclass PandasCSVReadStrategy(CSVReadStrategy):\n    def __init__(self):\n        self.name = \"PandasCSVReadStrategy\"\n    def read_lines(self, file_path, n_lines, **kwargs):\n        df = pd.read_csv(file_path, nrows=n_lines, **kwargs)\n        return df\n    \nclass OSCSVReadStrategy(CSVReadStrategy):\n    def __init__(self):\n        self.name = \"OSCSVReadStrategy\"\n    def read_lines(self, file_path, n_lines):\n        lines = []\n        with open(file_path, 'r') as file:\n            for _ in range(n_lines):\n                line = file.readline()\n                if not line:\n                    break\n                lines.append(line)\n            file.close()\n        return lines\n\n#class PandasStringIOStrategy(CSVReadStrategy):\n#    def __init__(self):\n#        self.name = \"PandasStringIOStrategy\"\n#    \n#    def read_lines(self, file_path, n_lines, **kwargs):\n#        # Convert data to a StringIO object\n#        string_io = StringIO(file_path)\n#        # stringIO enforce n_lines\n#        data_lines = string_io.readlines()[:n_lines]\n#        # Read CSV from the StringIO object\n#        df = pd.read_csv(data_lines, nrows=n_lines, **kwargs)\n#        return df\n    \nclass PySparkCSVReadStrategy(CSVReadStrategy):\n    def __init__(self):\n        self.name = \"PySparkCSVReadStrategy\"\n        self.spark = SparkSession.builder.appName(\"CSVReader\").getOrCreate()\n\n    def read_lines(self, file_path, n_lines, schema=None, **kwargs):\n        if 'new_schema' in kwargs:\n            new_schema = kwargs.get('new_schema', None)\n            df = self.spark.read.options(header=True, **kwargs).schema(new_schema).csv(file_path).limit(n_lines)\n        else:\n            df = self.spark.read.options(header=True,sampling_ratio=0.005, infer_schema=True, **kwargs).csv(file_path).limit(n_lines)\n            \n        return df\n\nclass PolarsCSVReadStrategy(CSVReadStrategy):\n    def __init__(self):\n        self.name = \"PolarsCSVReadStrategy\"\n    def read_lines(self, file_path, n_lines, **kwargs):        \n        # Read CSV using Polars\n        df = pl.read_csv(file_path, n_rows=n_lines, **kwargs)\n        \n        return df\n    \nclass DaskCSVReadStrategy(CSVReadStrategy):\n    def __init__(self):\n        self.name = \"DaskCSVReadStrategy\"\n    def read_lines(self, file_path, n_lines, **kwargs):\n        df = dd.read_csv(file_path, **kwargs).head(n=n_lines)\n        return df\n\nclass DatatableCSVReadStrategy(CSVReadStrategy):\n    def __init__(self):\n        self.name = \"DatatableCSVReadStrategy\"\n    def read_lines(self, file_path, n_lines, **kwargs):\n        if(kwargs==None):\n            df = dtt.fread(cmd = f\"\"\"head -n {n_lines} {file_path} {kwargs}\"\"\")\n        else:\n            df = dtt.fread(cmd = f\"\"\"head -n {n_lines} {file_path}\"\"\")\n        return df\n\nclass ModinPandasCSVReadStrategy(CSVReadStrategy):\n    def __init__(self):\n        self.name = \"ModinPandasCSVReadStrategy\"\n    def read_lines(self, file_path, n_lines, **kwargs):\n        df = modpd.read_csv(file_path, nrows=n_lines,header=0, **kwargs)\n        return df\n    \n###################################################\n#         DEFINE STRATEGIES                       #\n###################################################\nclass RSA:\n    def __init__(self, strategy: CSVReadStrategy):\n        self.strategy = strategy\n        \n    def all_strategies(self):\n        # Define a list of strategies\n        list_of_strategies = [\n            PandasCSVReadStrategy(),\n            OSCSVReadStrategy(),\n            #PandasStringIOStrategy(),\n            PySparkCSVReadStrategy(),\n            PolarsCSVReadStrategy(),\n            DaskCSVReadStrategy(),\n            DatatableCSVReadStrategy(),\n            ModinPandasCSVReadStrategy()\n        ]\n        return list_of_strategies\n\n    def set_strategy(self, strategy: CSVReadStrategy):\n        self.strategy = strategy\n\n    def read_lines(self, file_path, n_lines, **kwargs):\n        return self.strategy.read_lines(file_path, n_lines, **kwargs)\n    \n    def create_store(self):\n        self._store = {}\n        # Loop through each strategy\n        for st in self.all_strategies():\n            print(\"creating strategy in dictionary: \", st.name)\n            # Initialize a dictionary to store the results for the current strategy\n            self._store[str(st.name)] = {\n                \"time_taken\": [],\n                \"n_lines\": [],\n                \"kwargs\": None\n            }\n        return \"store was created. access it with rsa._store\"\n\n    @staticmethod\n    def help():\n        msg = f\"\"\"\nWelcome to Read Strategy Algorithm (RSA)\n\nThis algorithm helps in reading CSV files using different strategies. \nYou can choose from the following strategies:\n1. PandasCSVReadStrategy (Pandas)\n2. OSCSVReadStrategy (OS)\n~3. PandasStringIOStrategy (Pandas IO)~\n4. PySparkCSVReadStrategy (Pyspark)\n5. PolarsCSVReadStrategy (Polars)\n6. DaskCSVReadStrategy (Dask)\n7. DatatableCSVReadStrategy (datatable)\n8. ModinPandasCSVReadStrategy (modin)\n\nTo use this algorithm:\n1. Instantiate RSA with your desired strategy.\n2. Use the 'set_strategy' method to change the strategy if needed.\n3. Use the 'read_lines' method to read CSV files using the selected strategy.\n\nExample:\nrsa = RSA(PandasCSVReadStrategy())\nrsa.set_strategy(PandasCSVReadStrategy())\nrsa.read_lines('example.csv',n_lines=100)\n\"\"\"\n        return msg","metadata":{"execution":{"iopub.status.busy":"2024-06-09T15:34:20.390831Z","iopub.execute_input":"2024-06-09T15:34:20.391338Z","iopub.status.idle":"2024-06-09T15:34:20.423412Z","shell.execute_reply.started":"2024-06-09T15:34:20.391306Z","shell.execute_reply":"2024-06-09T15:34:20.421971Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(RSA.help())","metadata":{"execution":{"iopub.status.busy":"2024-06-09T15:34:20.90063Z","iopub.execute_input":"2024-06-09T15:34:20.901079Z","iopub.status.idle":"2024-06-09T15:34:20.907234Z","shell.execute_reply.started":"2024-06-09T15:34:20.901045Z","shell.execute_reply":"2024-06-09T15:34:20.9061Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"rsa = RSA(PandasCSVReadStrategy())","metadata":{"execution":{"iopub.status.busy":"2024-06-09T15:34:21.9703Z","iopub.execute_input":"2024-06-09T15:34:21.971083Z","iopub.status.idle":"2024-06-09T15:34:21.976212Z","shell.execute_reply.started":"2024-06-09T15:34:21.971045Z","shell.execute_reply":"2024-06-09T15:34:21.974992Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"rsa.create_store()","metadata":{"execution":{"iopub.status.busy":"2024-06-09T15:34:22.523657Z","iopub.execute_input":"2024-06-09T15:34:22.524089Z","iopub.status.idle":"2024-06-09T15:34:22.536925Z","shell.execute_reply.started":"2024-06-09T15:34:22.524059Z","shell.execute_reply":"2024-06-09T15:34:22.535531Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"rsa._store","metadata":{"execution":{"iopub.status.busy":"2024-06-09T15:34:23.371365Z","iopub.execute_input":"2024-06-09T15:34:23.371792Z","iopub.status.idle":"2024-06-09T15:34:23.382017Z","shell.execute_reply.started":"2024-06-09T15:34:23.371758Z","shell.execute_reply":"2024-06-09T15:34:23.380788Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# first read 500 lines (small sampling)","metadata":{}},{"cell_type":"code","source":"n_lines = 500\n\nfor st in rsa.all_strategies():\n    print(f\"executing strategy {st.name}...\")\n    \n    t = now()\n    \n    rsa.set_strategy(st)\n\n    kwargs = rsa._store[str(st.name)][\"kwargs\"]\n    \n    if(kwargs != None):\n        data = rsa.read_lines(files[\"train.csv\"], n_lines = n_lines, **kwargs)\n    else:\n        data = rsa.read_lines(files[\"train.csv\"], n_lines = n_lines)\n\n    dt = now() - t\n        \n    rsa._store[str(st.name)][\"time_taken\"] = dt.total_seconds()\n    rsa._store[str(st.name)][\"n_lines\"] = n_lines\n    rsa._store[str(st.name)][\"data\"] = data","metadata":{"execution":{"iopub.status.busy":"2024-06-09T15:34:24.84326Z","iopub.execute_input":"2024-06-09T15:34:24.844343Z","iopub.status.idle":"2024-06-09T15:34:28.731776Z","shell.execute_reply.started":"2024-06-09T15:34:24.844303Z","shell.execute_reply":"2024-06-09T15:34:28.7309Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"for k in rsa._store.keys():\n    print(k)\n    print(rsa._store[k][\"time_taken\"])\n    print(\"type of data:\", type(rsa._store[k][\"data\"]))\n    print(\"-\"*20)","metadata":{"execution":{"iopub.status.busy":"2024-06-09T15:34:28.733204Z","iopub.execute_input":"2024-06-09T15:34:28.733778Z","iopub.status.idle":"2024-06-09T15:34:28.740943Z","shell.execute_reply.started":"2024-06-09T15:34:28.733748Z","shell.execute_reply":"2024-06-09T15:34:28.739437Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from pyspark.sql.types import StructType, StructField, FloatType\n\n# Assume rsa is your dataframe\ndf = rsa._store[\"PySparkCSVReadStrategy\"][\"data\"]\n\n# Get the schema\nschema = df.schema\n\n# Create a new schema with desired changes\nnew_schema = StructType([\n    StructField(field.name, FloatType() if field.name != \"sample_id\" else field.dataType, field.nullable)\n    for field in schema\n])","metadata":{"execution":{"iopub.status.busy":"2024-06-09T15:34:30.866702Z","iopub.execute_input":"2024-06-09T15:34:30.867146Z","iopub.status.idle":"2024-06-09T15:34:30.933667Z","shell.execute_reply.started":"2024-06-09T15:34:30.867104Z","shell.execute_reply":"2024-06-09T15:34:30.932442Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"rsa._store[\"PySparkCSVReadStrategy\"][\"kwargs\"] = \\\n    {\"lineSep\":\"\\n\",\"new_schema\":new_schema}","metadata":{"execution":{"iopub.status.busy":"2024-06-09T15:34:32.938715Z","iopub.execute_input":"2024-06-09T15:34:32.939151Z","iopub.status.idle":"2024-06-09T15:34:32.944836Z","shell.execute_reply.started":"2024-06-09T15:34:32.93912Z","shell.execute_reply":"2024-06-09T15:34:32.943408Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"n_lines = 500\n\nfor st in rsa.all_strategies():\n    print(f\"executing strategy {st.name}...\")\n    \n    t = now()\n    \n    rsa.set_strategy(st)\n\n    kwargs = rsa._store[str(st.name)][\"kwargs\"]\n    \n    if(kwargs != None):\n        data = rsa.read_lines(files[\"train.csv\"], n_lines = n_lines, **kwargs)\n    else:\n        data = rsa.read_lines(files[\"train.csv\"], n_lines = n_lines)\n\n    dt = now() - t\n        \n    rsa._store[str(st.name)][\"time_taken\"] = dt.total_seconds()\n    rsa._store[str(st.name)][\"n_lines\"] = n_lines\n    rsa._store[str(st.name)][\"data\"] = data","metadata":{"execution":{"iopub.status.busy":"2024-06-09T15:34:34.227716Z","iopub.execute_input":"2024-06-09T15:34:34.228136Z","iopub.status.idle":"2024-06-09T15:34:37.883943Z","shell.execute_reply.started":"2024-06-09T15:34:34.228106Z","shell.execute_reply":"2024-06-09T15:34:37.882882Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"\nfor st in rsa._store.keys():\n    x = rsa._store[st][\"n_lines\"]\n    y = rsa._store[st][\"time_taken\"]\n    print(x)\n    print(y)","metadata":{"execution":{"iopub.status.busy":"2024-06-09T15:34:37.88571Z","iopub.execute_input":"2024-06-09T15:34:37.88604Z","iopub.status.idle":"2024-06-09T15:34:37.893058Z","shell.execute_reply.started":"2024-06-09T15:34:37.886013Z","shell.execute_reply":"2024-06-09T15:34:37.891832Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import matplotlib.pyplot as plt\n\nst =list(rsa._store.keys())\n\nx = [rsa._store[strategy][\"n_lines\"] for strategy in st]\ny = [rsa._store[strategy][\"time_taken\"] for strategy in st]\n\n# Plotting\nplt.bar(st, y, color='blue')\n\n# Adding labels and title\nplt.xlabel('State')\nplt.ylabel('Time Taken')\nplt.title('Time Taken vs State when n_lines = 500')\n\n# Rotating x-axis labels for better readability\nplt.xticks(rotation=90)\n\n# Displaying the plot\nplt.show()\n","metadata":{"execution":{"iopub.status.busy":"2024-06-09T15:36:30.481121Z","iopub.execute_input":"2024-06-09T15:36:30.481683Z","iopub.status.idle":"2024-06-09T15:36:30.796663Z","shell.execute_reply.started":"2024-06-09T15:36:30.48164Z","shell.execute_reply":"2024-06-09T15:36:30.795429Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Results analysis\n\nLooking at the graph above, you might think that Dask has the worst performance between all of these strategies for reading CSV files\n\n**but you yould be WRONG**\n\nWhat is actually happening is that DASK reads the file in a particular way, that partitions the file into N parts\n\nI discovered that because I plotted this graph many times\n\nOnce a specific strategy is being too far apart from the other strategies in terms of performance, **I TROUBLESHOOT**\n\nThe things I look for when troubleshooting are the parameters of that library and its modules for CSV reading or file reading, and I look for any parameters that might affect the number of rows being read.\n\nOn pandas, I came accross some, including nrows, chunksize\n\nBut Dask only has a limiting function that works AFTER the partitioning of the file\n\nWhich Might turn it into the fastest way to read a CSV file yet!\n\nWe will go over other strategies and iterations to approve this possibility","metadata":{}},{"cell_type":"code","source":"import matplotlib.pyplot as plt\n\nst =list(rsa._store.keys())\nst = [item for item in st if item != \"DaskCSVReadStrategy\"]\n\nx = [rsa._store[strategy][\"n_lines\"] for strategy in st]\ny = [rsa._store[strategy][\"time_taken\"] for strategy in st]\n\n# Plotting\nplt.bar(st, y, color='blue')\n\n# Adding labels and title\nplt.xlabel('State')\nplt.ylabel('Time Taken')\nplt.title('Time Taken vs State when n_lines = 500')\n\n# Rotating x-axis labels for better readability\nplt.xticks(rotation=90)\n\n# Displaying the plot\nplt.show()\n","metadata":{"execution":{"iopub.status.busy":"2024-06-09T15:36:34.217016Z","iopub.execute_input":"2024-06-09T15:36:34.217407Z","iopub.status.idle":"2024-06-09T15:36:34.527558Z","shell.execute_reply.started":"2024-06-09T15:36:34.217379Z","shell.execute_reply":"2024-06-09T15:36:34.526424Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def time_execution_analysis_between_strategies(m, s):\n    \"\"\"\n        Creates a dictionary of data for data analysis on best strategy to read a CSV\n        \n        m = Max number of ROWS to read from files\n        \n        s = Steps, in number of ROWS to iterate over to get the dat\n        \n        example:\n        \n        m = 1500\n        s = 500\n        \n        starts at reading 500 rows\n        reads every 500 rows:\n        - 500\n        - 1000\n        - 1500\n        up to 1500\n        \n        returns: strategy_data = dictionary datatype\n                    {strategy.name: {'x': [500, 1000, 1500],'y': [0.160281, 0.325411, 0.453121]},\n    \"\"\"\n    # Initialize a dictionary to store x and y values for each strategy\n    strategy_data = {st.name: {'x': [], 'y': []} for st in rsa.all_strategies()}\n\n    # max N of counter\n    max_exc = m / s\n    # Number of rows + steps\n    ms = m + s\n    \n    for n_lines in range(s,ms, s):\n        c_exc = n_lines / s\n\n        for st in rsa.all_strategies():\n            if(\n                (c_exc <= max_exc and st.name != \"DaskCSVReadStrategy\"\n                   or c_exc == max_exc and st.name == \"DaskCSVReadStrategy\")\n            ):\n                #print(f\"executing strategy {st.name}...\")\n\n                t = now()\n\n                rsa.set_strategy(st)\n\n                kwargs = rsa._store[str(st.name)][\"kwargs\"]\n\n                if(kwargs != None):\n                    data = rsa.read_lines(files[\"train.csv\"], n_lines = n_lines, **kwargs)\n                else:\n                    data = rsa.read_lines(files[\"train.csv\"], n_lines = n_lines)\n\n                del data\n\n                dt = now() - t\n\n                rsa._store[str(st.name)][\"time_taken\"] = dt.total_seconds()\n                rsa._store[str(st.name)][\"n_lines\"] = n_lines\n                #rsa._store[str(st.name)][\"data\"] = data\n\n                # Append values to the strategy-specific lists\n                strategy_data[st.name]['x'].append(n_lines)\n                strategy_data[st.name]['y'].append(dt.total_seconds())\n\n    return strategy_data","metadata":{"execution":{"iopub.status.busy":"2024-06-09T15:20:21.195044Z","iopub.execute_input":"2024-06-09T15:20:21.19537Z","iopub.status.idle":"2024-06-09T15:20:21.209048Z","shell.execute_reply.started":"2024-06-09T15:20:21.195344Z","shell.execute_reply":"2024-06-09T15:20:21.207681Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"strat = time_execution_analysis_between_strategies(m=1500,s=500)","metadata":{"execution":{"iopub.status.busy":"2024-06-09T15:20:21.210776Z","iopub.execute_input":"2024-06-09T15:20:21.211218Z","iopub.status.idle":"2024-06-09T15:20:28.139331Z","shell.execute_reply.started":"2024-06-09T15:20:21.211186Z","shell.execute_reply":"2024-06-09T15:20:28.138071Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"strat","metadata":{"execution":{"iopub.status.busy":"2024-06-09T15:20:28.14073Z","iopub.execute_input":"2024-06-09T15:20:28.141073Z","iopub.status.idle":"2024-06-09T15:20:28.150415Z","shell.execute_reply.started":"2024-06-09T15:20:28.141046Z","shell.execute_reply":"2024-06-09T15:20:28.14929Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Plotting the results\nplt.figure(figsize=(10, 6))\n\nfor st_name, data in strat.items():\n    plt.plot(data['x'], data['y'], label=st_name)\n\nplt.xlabel('Number of Lines')\nplt.ylabel('Time Taken (seconds)')\nplt.title('Time Taken for Different Strategies')\nplt.legend()\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2024-06-09T15:20:28.152297Z","iopub.execute_input":"2024-06-09T15:20:28.152741Z","iopub.status.idle":"2024-06-09T15:20:28.562222Z","shell.execute_reply.started":"2024-06-09T15:20:28.152703Z","shell.execute_reply":"2024-06-09T15:20:28.561032Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# provides 10 iterations\n#Million_1 = 1000000\n#H_Thousand = 100000\n\n#strat = time_execution_analysis_between_strategies(m=Million_1,s=H_Thousand)","metadata":{"execution":{"iopub.status.busy":"2024-06-09T15:20:28.563827Z","iopub.execute_input":"2024-06-09T15:20:28.564284Z","iopub.status.idle":"2024-06-09T15:20:28.569481Z","shell.execute_reply.started":"2024-06-09T15:20:28.564244Z","shell.execute_reply":"2024-06-09T15:20:28.568324Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Plotting the results\n#plt.figure(figsize=(10, 6))\n\n#for st_name, data in strat.items():\n#    plt.plot(data['x'], data['y'], label=st_name)\n\n#plt.xlabel('Number of Lines')\n#plt.ylabel('Time Taken (seconds)')\n#plt.title('Time Taken for Different Strategies')\n#plt.legend()\n#plt.show()","metadata":{"execution":{"iopub.status.busy":"2024-06-09T15:20:28.570774Z","iopub.execute_input":"2024-06-09T15:20:28.571167Z","iopub.status.idle":"2024-06-09T15:20:28.580436Z","shell.execute_reply.started":"2024-06-09T15:20:28.571139Z","shell.execute_reply":"2024-06-09T15:20:28.57946Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# CSV Strategies V5 Analysis : 4 strategies\n\nAnalysing READS for CSV file\n\n- Pandas\n- OS\n- PySpark\n    (its performance here is probably related to pyspark not having any action such as querying, saving data, etc.)\n- Polars\n\n    - y = time taken (seconds)\n    - x = number of lines (^6)\n        - 100.000\n        - 200.000\n        - ...\n        - 1.000.000\n        \nTime taken: 6 Hours, 26 minutes\n    Iterations: 40 (4 strategies, 10 row Iterations)","metadata":{}},{"cell_type":"markdown","source":"![alt text](https://github.com/EmaoqFilho-NTTData/csv_strategy/blob/main/strategies_v5.png?raw=true \"Title\")","metadata":{}},{"cell_type":"markdown","source":"# WIP...","metadata":{}}]}