Data Science Portfolio: Crop Productivity and Market Conditions

Hello traveler! Welcome to our humble island. Looks like you are one of the researchers from the mainland.

Yes! We are actually sent here to help understand the island anomaly.

The island anomaly... market prices fluctuates and we barely breakeven, and our farmers are struggling. What might be causing this?

That's what we're here to answer. While food crop production is mainly affected by different natural conditions, we hypothesize that there might be other forces at play. The invisible hand of the market, often overlooked, may have the ability to influence our agricultural productivity.

I see. If you will, I have a humble request.
In the upcoming city council meeting, I hope you can present some of your findings to help us common folks understand and manage this anomaly.

As a scholar, it is in your best interest to seek evidence and explore the impact of economic forces and policies to agricultural food production using different statistical techniques and machine learning tools. You also intend to uncover patterns that could help resolve the current agricultural crisis. Your ultimate goal is help achieve food security and promote sustainable agriculture for this island and the world beyond. You soon left the town to discover more about this island and their troubles. Press the exit button to continue with your journey.

After the pandemic-driven recession in 2020, the island (i.e., the Philippines) is still experiencing economic and food crisis.

Agriculture plays a huge role in the lives of Filipinos. However, our nation still struggles when it comes to food security. Shortage of food supply coming from the agricultural sector is common. Leading causes are rampant hoarding, inadequate government policies, and unforeseen natural crises. Sharp inflation in food prices are also commonplace. One can recall the onion and chili fiascos in recent years.

In 2023, to combat rising rice prices nationwide, President Marcos Jr. declared a rice price ceiling that fueled further economic insecurity for rice farmers and retailers. The policy caused some short term stability, but eventually dominoed into added pressure in producing food crops and in selling them in the market. Related to this are the cases of vegetables such as cabbages that are being left to rot or forcibly sold at a loss due to an interesting combination of oversupply and uneven demand across the archipelago, and across producers and consumers. This is food that could be bought by the government and reallocated to starving families. With the ever increasing population and unpredictable crises, the capacity to produce food for the population is hampered. Sustainable agriculture and food production is needed now more than ever.

References:

World Food Programme

Department of Agriculture

Trinh Nguyen - Asia Program

Ramon Royandoyan - Nikkei Asia

HelpAge International

This is to answer the elusive question: How do market conditions correlate with food crops productivity? NULL HYPOTHESIS: Economic factors does not have any correlation with food crop productivity in the several regions of the country ALTERNATIVE HYPOTHESIS: Food crops production in the country are correlated by some economic factors ACTION PLAN: Analyze and interpret several economic factors and their relation to different food crops production. Additionally, formulate models for important agricultural indicators. You left the store to continue your journey. Press the exit button to continue with your journey.

The team gathered most of the required data from the database maintained by the Philippine Statistical Authority (PSA). Various datasets were collected with varying temporal (i.e. Period or Datetime) granularity and range (with a unified range of 1994-2023). However, due to granularity issues, only the annualized range 2012-2023 were used for hypothesis testing, but the entire 1994-2023 range is still used for machine learning as detailed later.

To autonomously gather the data from PSA, the team developed scrapers using Mechanical Soup.

Due to the limitations of the scraper, the team also developed a program to compile fragmented datasets into one with the help of Pandas.

You begin to sanitize and standardize the data you've collected:

For proper format consistency, Year and Months column were properly arranged in chronological order. Additionally, textcases for some elements were fixed.
Converted and validated all ".." and 0 elements to blank representations for easy NA representation.
Removed columns such as "Annual" and "Average" that can be derived from current data to reduce data clutter.
Region-Province relationships were derived and kept into a separate file, and multiple region name corrections had to be done (e.g. ARMM vs BARMM).
Aside from geolocation naming issues, bad and non-standard commodity names and period names (e.g. abbreviated months vs. full months) also had to be fixed.
Period values standardized to the range 1994-2023, either Yearly, Quarterly, or Monthly (Yearly is where most the datasets intersect, hence it is used for hypothesis testing). Reduced number of features however are used for machine learning, hence allowing 1994-2023 Quarterly and Monthly to be used.

The following cereal crops are the main focus of this study. Given are some fun facts about them.

Crops:
Rice - The country is the 8th largest rice producer in the world. However, rice is the Philippines' 3rd largest agricultural import, meaning our nation is still struggling with agricultural self-sufficiency. Corn - This is the second most important crop in the country. Sweet corn is a staple treat for Filipinos and its grains are also an important feed for livestock.

The following are the collected agricultural and socioeconomic indicators to be used as features.

Agricultural Indicators:
Stocks - Sum of household, commercial, and NFA commodity stocks./p2> Volume - Total quantity of a crop produced and harvested, including portions damaged, stolen, given away, consumed, or reserved. Area Harvested - Actual area from which crops are harvested, excluding totally damaged areas. Self-Sufficiency Ratio - Measures the extent to which a country's domestic production meets its domestic utilization. Import-Dependency Ratio - Measures the extent to which a country relies on imports to meet its domestic consumption needs for crops. SU Gross Supply - Total quantity of food available before any use, including beginning stocks, production, and imports. UT Consumable - Amount of food commodity available in its original (unprocessed) form for human consumption.
SU = Supply Utilization, UT = Utilization Total

Socioeconomic Indicators:
Farmgate Price - Price received by farmers for their crops (excludes transport and marketing costs). Retail Price - Price at which goods are sold to regular consumers. Wholesale Price - Price at which goods are sold in large quantities from producers or distributors to retailers. CPI All Income - Change in average retail prices of a fixed basket of goods and services purchased by all households compared to a base year CPI Bottom 30 - Change in average retail prices of a fixed basket of goods and services purchased by the lowest income 30% of households compared to a base year Net Returns - Commodity total production costs (cash and non-cash) minus total revenue from commodity Inflation Rate - Percentage change in the average price level of goods and services over a specific period, compared to the previous period or year NCR RPI - Food Retail Price Index at the National Capital Region
RPI = Retail Price Index, CPI = Consumer Price Index

This section shows relevant and interactive visualization from the data you gathered pertaining to potential relationships between economic indicators and agricultural indicators for selected crops.

Switch Food Crop

The graph shows rapid flactuation on both volume of production and retail price for rice. The sudden drop on volume of production in 2016 can be attributed to prolonged El Nino.

The correlation matrix for rice indicates strong correlations between Inflation Rate and several agricultural indicators. Same with NCR RPI. However, most feature pairs appear to show weak or no correlation. We'll see in the hypothesis testing the numbers to verify these.

This section shows focused graphs discussing some relevant and interesting points you found in the data.

The focused graph shows an interesting proportional relationship between stocks of both rice and corn. Visually, it seems that stocks tend go down when there is a spike or increased in inflation rate. This may signify an inverse relationship between stocks and inflation rate.

Investigating time series of rice wholesale price leads to interesting peaks that may be attributed to some economic and agricultural policies. This signifies the rice insecurity the country deals with.

References:

2014 Rice Hoarding Issue

2016 4th Quarter Spike on Rice Price

2018 Rice Crisis

2019 Enactment of Rice Tariffication Law

2020 Pandemic Rice Price Freeze

2023 High Rice Inflation

To answer the problem, you will be testing your hypotheses at a standard 0.05 level of significance.

To determine whether or not a significant relationship exists between the Market Indicators and Agricultural Indicators you collected, you used Pearson Correlation Coefficient. This allows you to measure the strength of their potential correlations. You used all observations for both rice and corn for this.

You observed several interesting relationships, such as the fact that higher Self-Sufficiency Ratio relates to lower food prices in the capital (NCR RPI), and the inverse is true. This is just one of your many exciting realizations.

In analyzing if significant difference exists between rice and corn in terms of both agricultural and socioeconomic indicators when grouped by regions, you used two-way t-test. The sample graph helps in visualizing this idea.

Wow! You realize that certain food crops differ in profitability, abundance, and costs, among others, in certain regions.

You feel like you're getting close to unraveling the island's mysteries.

Result 1: The significant correlations and regional differences highlight that some market conditions, such as inflation rate, play a role in determining agricultural productivity. Generally, a low price on grains signifies a high self-sufficiency ratio for the agricultural sector.

Result 2: Despite having growing stocks on grains every year, prices still continue to grow. This can be attributed to the continuous growth of the population and increasing reliance on much more expensive imported grains.

Result 3: Policies attempting to stabilize rice and corn supplies only work for a short period of time. This highlights the weakness of the island to find solutions for sustainable agriculture and a stable market.

Your EDA has given you more confidence that more patterns in your data can be learned. That is, there are more nuances embedded in the data than meets the (statistical) eye.

During this portion of your presentation, you showcase your ML experiments from start to finish to supplement your EDA findings and to better show the use of ML models for time-series analysis and forecasting (e.g. in predicting how the island's agriculture might perform in the near future). "Give a man a bowl of rice and he eats for a day, give a man a love for data science and he can model agricultural trends to get plenty more rice."

You focus your analysis on rice and its two agricultural indicators that are most important for the people: volume of rice production and rice stocks.

By preprocessing data and getting exogenous variables (e.g. the market indicators) that you deemed helpful in your model's prediction, you are now prepared to split the data into training and testing sets.

To actually find out which regression model suits best with the given data, you analyze their performance or the quality of their predictions using Mean Absolute Percentage Error (MAPE) and the Coefficient of Determination (R2). MAPE is to minimized while R2 is to be maximized while performing other validation to avoid overfitting.

After optimization and various experimentations on different models and parameter sets with respect to MAPE and R2, you determine that Elastic Net CV is the best model for predicting the (quarterly) volume of rice production and that Random Forest Regressor is the best model for predicting (monthly) rice stocks. The validation plots of these models are shown below.

With the models now validated, you can now use this to predict the agricultural productivity of the Island in the near future. The future predictions begins after the green vertical lines.

You found that the volume of rice production on the island would likely stay the same in the near future, while the rice stocks on the island, despite dipping for a short period, can eventually rebound. You share the news of better days ahead to the celebration of the islanders. The anomaly is in fact not an anomaly, but a pattern to be understood. Through your experiments, you also found out that various combinations of the market indicators have very minimal impact on the quality of the predictions, hence you are more keen to acknowledge that the answer to your primary research question is that the tested market indicators are only weakly correlated to agricultural productivity.

The people are grateful to you for giving them these models as these will help them people make an informed decision about future policies and redirect their prospects towards a more sustainable agriculture. Moreover, this will help them ascertain how their country's government policies are affecting their agriculture by comparing predicted values with actual values.

Before the meeting ended, you adviced them to retune the models from time to time to keep their accuracy.

This overall highlights that the island is not self-sufficient in terms of both rice and corn production. Analysis suggests that being self-sufficient is a key to lower domestic prices on food crops and can stabilize market conditions.

In relation to this, the island is currently experiencing inflation on rice prices. This can be attributed to the fact that the local production is low and expensive sanctions on cereal crops imports. Furthermore, heavy reliance on imports also introduces competition to domestic producers which harms the ability of the island to truly sustain itself.

All this trickles down to the realization that the current laws and policies of the government towards food sustainability is not enough. Analysis shows that food crops are affected by some market factors and policies currently imposed. With continuous population growth, policymakers should consider focusing on strategies pertaining to sustainable agriculture and market stability.

Additionally, regional patterns and correlations between agricultural and market indicators can be observed, ranging from weak to strong, with machine learning models revealing deeper insights.

The limitations of the project comes from the dataset itself. Certain data points or indicators are missing or not up-to-date, it could affect the comprehensiveness and accuracy of the analysis. Furthermore, certain regions may have more comprehensive data compared to others, leading to unequal representation and can affect the generalizability of the findings.

For future endeavors, you recommend the following:

A comprehensive and updated dataset of different economic and crop production from different regions of the island can help narrow down definitive conclusions about the current agricultural and economic condition of the island.
A more targeted analysis technique that is appropriate for economic analysis could bring out more information on the effects of market conditions on crop production.
This topic could be continued using other agricultural products such as livestock and other food products that could give new insights on this problem.
Encourage support for local farmers and their products by making informed decisions when it comes to voting those who will run the island.

We are the Baesians and this is our project, the Baesian Cropability. The project is an analysis of food crop production in relation to different market conditions and policies.

As a researcher, equipped with data science techniques, your aim is to understand and help solve the troubles of this island in crisis.

We are the Baesians and this is our project, the Baesian Cropability. The project is an analysis of food crop production in relation to different market conditions and policies.

As a researcher, equipped with data science techniques, your aim is to understand and help solve the troubles of this island in crisis.

The farmer welcomes you to Cereaulean Island

You accepted the farmer's request

You picked up the latest edition of the newspaper

You started forming hypotheses and potential solutions

You visited a local tool shop

It is time to plan out on how you will organize your data

You visited the local library

You were invited to talk in the council meeting today

This led you to conclude that...

The team appreciates you spending your time to learn.