Note: this demo is no longer active.
In this 3-part post, I'm going to show how you can take CIA World Factbook data and use it for your own purposes on Microsoft Azure.
Here in Part 1, we'll create the back end by retrieving the data and storing it in a Cosmos DB using Azure Durable Functions. In Part 2, we'll create the front end including an API and a web site for browsing, querying, and reporting on the data. In Part 3, we'll take the front-end further with analytics and charting.
What We're Building
Let's take a look at the end result. When we're done with Part 1, we will bring an extensive amount of data on countries of the world into a Cosmos DB database, which we'llbe able to query in numerous ways.
Querying Country Data in Cosmos DB
We'll also be putting the JSON record for each country in blob storage, along with its flag and map image.
JSON Country Data and Flag & Map images in Blob Storage
We'll handle both the initial data capture, and subsequent updates, using Azure Durable Functions written in C#/NET Core. Because the single large JSON download contains data for 260 countries, we'll be using a fan-out / fan-in pattern to store the data, Map/Reduce style.
CIA World Factbook: Comprehensive Country Data
The US Central Intelligence Agency publishes an alamanc-style reference on the countries of the world known as the CIA World Factbook. There's a wealth of data, and you can learn a lot on the site. I urge you to explore it and drill into the detail.
The World Factbook Site
What if you want to use this data in a different way? Perhaps you want to expose the data in a different user interface; or combine it with other data in your possession; or analyze it; or create infographics.
Conveniently, the factbook data is in the public domain: "The Factbook is in the public domain. Accordingly, it may be copied freely without permission of the Central Intelligence Agency (CIA). The official seal of the CIA, however, may NOT be copied without permission as required by the CIA Act of 1949 (50 U.S.C. section 403m). Misuse of the official seal of the CIA could result in civil and criminal penalties."
So, we're free to take this data and use it. Naturally, you should attribute the data to its source and adhere to the terms of use. Unfortunately, the data isn't directly provided in a machine-friendly format. Happily, a gentleman named Ian Coleman has remedied that with a JSON edition of the country data. that is hosted on GitHub and updated weekly. This is what we'll be capturing to make our own copy of the data.
Data Repository
If we're going to host this data in Azure, what's the best way to store it? First off, let's take a look at the data itself we'll be getting. A single 14MB download contains the data for 260 countries all in one JSON file. Here's an excerpt of the country data for just one country, Armenia: "armenia": {
"data": {
"name": "Armenia",
"introduction": {
"background": "Armenia prides itself on being the first nation to formally adopt Christianity (early 4th century). Despite periods of autonomy, over the centuries Armenia came under the sway of various empires including the Roman, Byzantine, Arab, Persian, and Ottoman. During World War I in the western portion of Armenia, the Ottoman Empire instituted a policy of forced resettlement coupled with other harsh practices that resulted in at least 1 million Armenian deaths. The eastern area of Armenia was ceded by the Ottomans to Russia in 1828; this portion declared its independence in 1918, but was conquered by the Soviet Red Army in 1920.\nArmenian leaders remain preoccupied by the long conflict with Azerbaijan over Nagorno-Karabakh, a primarily Armenian-populated region, assigned to Soviet Azerbaijan in the 1920s by Moscow. Armenia and Azerbaijan began fighting over the area in 1988; the struggle escalated after both countries attained independence from the Soviet Union in 1991. By May 1994, when a trilateral cease-fire between Armenia, Azerbaijan, and Nagorno-Karabakh took hold, ethnic Armenian forces held not only Nagorno-Karabakh but also seven surrounding regions - approximately 14 percent of Azerbaijan’s territory. The economies of both sides have been hurt by their inability to make substantial progress toward a peaceful resolution.\nTurkey closed the common border with Armenia in 1993 in support of Azerbaijan in its conflict with Armenia over control of Nagorno-Karabakh and surrounding areas, further hampering Armenian economic growth. In 2009, Armenia and Turkey signed Protocols normalizing relations between the two countries, but neither country ratified the Protocols, and Armenia officially withdrew from the Protocols in March 2018. In January 2015, Armenia joined Russia, Belarus, and Kazakhstan as a member of the Eurasian Economic Union. In November 2017, Armenia signed a Comprehensive and Enhanced Partnership Agreement (CEPA) with the EU. In spring 2018, Serzh SARGSIAN of the Republican Party of Armenia (RPA) stepped down and Civil Contract party leader Nikol PASHINYAN became prime minister."
},
"geography": {
"location": "Southwestern Asia, between Turkey (to the west) and Azerbaijan; note - Armenia views itself as part of Europe; geopolitically, it can be classified as falling within Europe, the Middle East, or both",
"geographic_coordinates": {
"latitude": {
"degrees": 40,
"minutes": 0,
"hemisphere": "N"
},
"longitude": {
"degrees": 45,
"minutes": 0,
"hemisphere": "E"
}
},
"map_references": "Asia",
"area": {
"total": {
"value": 29743,
"units": "sq km"
},
"land": {
"value": 28203,
"units": "sq km"
},
"water": {
"value": 1540,
"units": "sq km"
},
"global_rank": 143,
"comparative": "slightly smaller than Maryland"
},
"land_boundaries": {
"total": {
"value": 1570,
"units": "km"
},
"border_countries": [
{
"country": "Azerbaijan",
"border_length": {
"value": 996,
"units": "km"
}
},
{
"country": "Georgia",
"border_length": {
"value": 219,
"units": "km"
}
},
{
"country": "Iran",
"border_length": {
"value": 44,
"units": "km"
}
},
{
"country": "Turkey",
"border_length": {
"value": 311,
"units": "km"
}
}
]
},
"coastline": {
"value": 0,
"units": "km",
"note": "landlocked"
},
"climate": "highland continental, hot summers, cold winters",
"terrain": "Armenian Highland with mountains; little forest land; fast flowing rivers; good soil in Aras River valley",
"elevation": {
"mean_elevation": {
"value": 1792,
"units": "m",
"note": "400 m"
},
"lowest_point": "Debed River",
"4090_highest_point": "Aragats Lerrnagagat'"
},
"natural_resources": {
"resources": [
"gold",
"copper",
"molybdenum",
"zinc",
"bauxite"
]
},
"land_use": {
"by_sector": {
"agricultural_land_total": {
"value": 59.7,
"units": "%"
},
"arable_land": {
"value": 15.8,
"units": "%",
"note": "/"
},
"permanent_crops": {
"value": 1.9,
"units": "%",
"note": "/"
},
"permanent_pasture": {
"value": 42,
"units": "%"
},
"forest": {
"value": 9.1,
"units": "%"
},
"other": {
"value": 31.2,
"units": "%"
}
},
"date": "2014"
},
"irrigated_land": {
"value": 2740,
"units": "sq km",
"date": "2012"
},
"population_distribution": "most of the population is located in the northern half of the country; the capital of Yerevan is home to more than five times as many people as Gyumri, the second largest city in the country",
"natural_hazards": [
{
"description": "occasionally severe earthquakes",
"type": "hazard"
},
{
"description": "droughts",
"type": "hazard"
}
],
"environment": {
"current_issues": [
"soil pollution from toxic chemicals such as DDT",
"deforestation",
"pollution of Hrazdan and Aras Rivers",
"the draining of Sevana Lich (Lake Sevan)",
"a result of its use as a source for hydropower",
"threatens drinking water supplies",
"restart of Metsamor nuclear power plant in spite of its location in a seismically active zone"
],
"international_agreements": {
"party_to": [
"Air Pollution",
"Biodiversity",
"Climate Change",
"Climate Change-Kyoto Protocol",
"Desertification",
"Environmental Modification",
"Hazardous Wastes",
"Law of the Sea",
"Ozone Layer Protection",
"Wetlands"
],
"signed_but_not_ratified": [
"Air Pollution-Persistent Organic Pollutants"
]
}
}
},
"people": {
"population": {
"total": 3038217,
"global_rank": 137,
"date": "2018-07-01"
},
"nationality": {
"noun": "Armenian(s)",
"adjective": "Armenian"
},
"ethnic_groups": {
"ethnicity": [
{
"name": "Armenian",
"percent": 98.1
},
{
"name": "Yezidi",
"percent": 1.2,
"note": "Kurd"
},
{
"name": "other",
"percent": 0.7
}
],
"date": "2011"
},
"languages": {
"language": [
{
"name": "Armenian",
"percent": 97.9,
"note": "official"
},
{
"name": "Kurdish",
"percent": 1,
"note": "spoken by Yezidi minority"
},
{
"name": "other",
"percent": 1
}
],
"note": "Russian is widely spoken",
"date": "2011"
},
"religions": {
"religion": [
{
"name": "Armenian Apostolic",
"percent": 92.6
},
{
"name": "Evangelical",
"percent": 1
},
{
"name": "other",
"percent": 2.4
},
{
"name": "none",
"percent": 1.1
},
{
"name": "unspecified",
"percent": 2.9
}
],
"date": "2011"
},
"age_structure": {
"0_to_14": {
"percent": 18.86,
"males": 303712,
"females": 269279
},
"15_to_24": {
"percent": 12.37,
"males": 195722,
"females": 179970
},
"25_to_54": {
"percent": 43.31,
"males": 640089,
"females": 675643
},
"55_to_64": {
"percent": 13.77,
"males": 192515,
"females": 225882
},
"65_and_over": {
"percent": 11.7,
"males": 142835,
"females": 212570
},
"date": "2018"
},
"dependency_ratios": {
"ratios": {
"total_dependency_ratio": {
"value": 44.4,
"units": "%"
},
"youth_dependency_ratio": {
"value": 28.7,
"units": "%"
},
"elderly_dependency_ratio": {
"value": 15.8,
"units": "%"
},
"potential_support_ratio": {
"value": 6.3,
"units": "%"
}
},
"date": "2015"
},
"median_age": {
"total": {
"value": 35.6,
"units": "years"
},
"male": {
"value": 33.9,
"units": "years"
},
"female": {
"value": 37.4,
"units": "years"
},
"global_rank": 78,
"date": "2018"
},
"population_growth_rate": {
"growth_rate": -0.25,
"global_rank": 213,
"date": "2018"
},
"birth_rate": {
"births_per_1000_population": 12.6,
"global_rank": 153,
"date": "2018"
},
"death_rate": {
"deaths_per_1000_population": 9.5,
"global_rank": 47,
"date": "2018"
},
"net_migration_rate": {
"migrants_per_1000_population": -5.7,
"global_rank": 194,
"date": "2017"
},
"population_distribution": "most of the population is located in the northern half of the country; the capital of Yerevan is home to more than five times as many people as Gyumri, the second largest city in the country",
"urbanization": {
"urban_population": {
"value": 63.1,
"units": "%",
"date": "2018"
},
"rate_of_urbanization": {
"value": 0.22,
"units": "%"
}
},
"major_urban_areas": {
"places": [
{
"place": "Yerevan",
"population": 1080000,
"is_capital": true
}
],
"date": "2018"
},
"sex_ratio": {
"by_age": {
"at_birth": {
"value": 1.12,
"units": "males/female"
},
"0_to_14_years": {
"value": 1.14,
"units": "males/female"
},
"15_to_24_years": {
"value": 1.06,
"units": "males/female"
},
"25_to_54_years": {
"value": 0.93,
"units": "males/female"
},
"55_to_64_years": {
"value": 0.84,
"units": "males/female"
},
"65_years_and_over": {
"value": 0.67,
"units": "males/female"
}
},
"total_population": {
"value": 0.94,
"units": "males/female"
},
"date": "2017"
},
"mothers_mean_age_at_first_birth": {
"age": 24.4,
"date": "2015"
},
"maternal_mortality_rate": {
"deaths_per_100k_live_births": 25,
"global_rank": 120,
"date": "2015"
},
"infant_mortality_rate": {
"total": {
"value": 12.3,
"units": "deaths_per_1000_live_births"
},
"male": {
"value": 13.7,
"units": "deaths_per_1000_live_births"
},
"female": {
"value": 10.7,
"units": "deaths_per_1000_live_births"
},
"global_rank": 110,
"date": "2018"
},
"life_expectancy_at_birth": {
"total_population": {
"value": 75.1,
"units": "years"
},
"male": {
"value": 71.8,
"units": "years"
},
"female": {
"value": 78.7,
"units": "years"
},
"global_rank": 113,
"date": "2018"
},
"total_fertility_rate": {
"children_born_per_woman": 1.64,
"global_rank": 177,
"date": "2018"
},
"contraceptive_prevalence_rate": {
"value": 57.1,
"units": "%",
"date": "2015"
},
"health_expenditures": {
"percent_of_gdp": 4.5,
"global_rank": 155,
"date": "2014"
},
"physicians_density": {
"physicians_per_1000_population": 2.8,
"date": "2014"
},
"hospital_bed_density": {
"beds_per_1000_population": 4.2,
"date": "2015"
},
"drinking_water_source": {
"improved": {
"urban": {
"value": 100,
"units": "percent of population"
},
"rural": {
"value": 100,
"units": "percent of population"
},
"total": {
"value": 100,
"units": "percent of population"
}
},
"unimproved": {
"urban": {
"value": 0,
"units": "percent of population"
},
"rural": {
"value": 0,
"units": "percent of population"
},
"total": {
"value": 0,
"units": "percent of population"
}
},
"date": "2015"
},
"sanitation_facility_access": {
"improved": {
"urban": {
"value": 96.2,
"units": "percent of population"
},
"rural": {
"value": 78.2,
"units": "percent of population"
},
"total": {
"value": 89.5,
"units": "percent of population"
}
},
"unimproved": {
"urban": {
"value": 3.8,
"units": "percent of population"
},
"rural": {
"value": 21.8,
"units": "percent of population"
},
"total": {
"value": 10.5,
"units": "percent of population"
}
},
"date": "2015"
},
"hiv_aids": {
"adult_prevalence_rate": {
"percent_of_adults": 0.2,
"global_rank": 91,
"date": "2017"
},
"people_living_with_hiv_aids": {
"total": 3400,
"global_rank": 121,
"date": "2017"
},
"deaths": {
"total": 200,
"date": "2017"
}
},
"adult_obesity": {
"percent_of_adults": 20.2,
"global_rank": 101,
"date": "2016"
},
"underweight_children": {
"percent_of_children_under_the_age_of_five": 2.6,
"global_rank": 106,
"date": "2016"
},
"education_expenditures": {
"percent_of_gdp": 2.8,
"global_rank": 151,
"date": "2016"
},
"literacy": {
"definition": "age 15 and over can read and write",
"total_population": {
"value": 99.7,
"units": "%"
},
"male": {
"value": 99.7,
"units": "%"
},
"female": {
"value": 99.6,
"units": "%"
},
"date": "2015"
},
"school_life_expectancy": {
"total": {
"value": 13,
"units": "years"
},
"male": {
"value": 13,
"units": "years"
},
"female": {
"value": 13,
"units": "years"
},
"date": "2015"
},
"youth_unemployment": {
"total": {
"value": 36.3,
"units": "%"
},
"male": {
"value": 29.5,
"units": "%"
},
"female": {
"value": 45.7,
"units": "%"
},
"global_rank": 18,
"date": "2016"
}
},
"government": {
"country_name": {
"conventional_long_form": "Republic of Armenia",
"conventional_short_form": "Armenia",
"local_long_form": "Hayastani Hanrapetut'yun",
"local_short_form": "Hayastan",
"former": "Armenian Soviet Socialist Republic, Armenian Republic",
"etymology": "the etymology of the country's name remains obscure; according to tradition, the country is named after Hayk, the legendary patriarch of the Armenians and the great-great-grandson of Noah; Hayk's descendant, Aram, purportedly is the source of the name Armenia"
},
"government_type": "parliamentary democracy; note - constitutional changes adopted in December 2015 transformed the government to a parliamentary system",
"capital": {
"name": "Yerevan",
"geographic_coordinates": {
"latitude": {
"degrees": 40,
"minutes": 10,
"hemisphere": "N"
},
"longitude": {
"degrees": 44,
"minutes": 30,
"hemisphere": "E"
}
},
"time_difference": {
"timezone": 4,
"note": "9 hours ahead of Washington, DC, during Standard Time"
},
"etymology": "name likely derives from the ancient Urartian fortress of Erebuni established on the current site of Yerevan in 782 B.C. and whose impresive ruins still survive"
},
"independence": {
"date": "1991-09-21",
"note": "from the Soviet Union"
},
"national_holidays": [
{
"name": "Independence Day",
"day": "21 September",
"original_year": "1991"
}
],
"constitution": {
"history": "previous 1915, 1978; latest adopted 5 July 1995 (2017)",
"amendments": "proposed by the president of the republic or by the National Assembly; passage requires approval by the president, by the National Assembly, and by a referendum with at least 25% registered voter participation and more than 50% of votes; constitutional articles on the form of government and democratic procedures are not amendable; amended 2005, 2007, 2008, last in 2015 (2017)",
"note": "a 2015 amendment, approved in December 2015 by a public referendum and effective for the 2017-18 electoral cycle, changes the government type from the current semi-presidential system to a parliamentary system"
},
"legal_system": "civil law system",
"international_law_organization_participation": [
"has not submitted an ICJ jurisdiction declaration",
"non-party state to the ICCt"
],
"citizenship": {
"citizenship_by_birth": "no",
"citizenship_by_descent_only": "at least one parent must be a citizen of Armenia",
"dual_citizenship_recognized": "yes",
"residency_requirement_for_naturalization": "3 years"
},
"suffrage": {
"age": 18,
"universal": true,
"compulsory": false
},
"executive_branch": {
"chief_of_state": "President Armen SARKISSIAN (since 9 April 2018)",
"head_of_government": "Prime Minister Nikol PASHINYAN (since 8 May 2018); First Deputy Prime Minister Ararat MIRZOYAN (since 11 May 2018)",
"cabinet": "Council of Ministers appointed by the prime minister",
"elections_appointments": "president indirectly elected by the National Assembly in 3 rounds if needed for a single 7-year term; election last held on 2 March 2018; prime minister elected by majority vote in 2 rounds if needed by the National Assembly; election last held on 8 May 2018",
"election_results": "Armen SARKISSIAN elected president in first round; note - Armen SARKISSIAN ran unopposed and won the Assembly vote 90-10; Nikol PASHINYAN elected prime minister in second round; note - Nikol PASHINYAN ran unopposed and won the Assembly vote 59-42",
"note": "Nikol PASHINYAN resigned his post (but stayed on as acting prime minister) on 16 October 2018 to force a snap election (held on 9 December 2018) in which his bloc won more than 70% of the vote; PASHINYAN was reappointed prime minister on 14 January 2019"
},
"legislative_branch": {
"description": "unicameral National Assembly (Parliament) or Azgayin Zhoghov (minimum 101 seats, currently 132; members directly elected in single-seat constituencies by proportional representation vote; members serve 5-year terms)",
"elections": "last held on 9 December 2018 (next elections to be held December 2023)",
"election_results": "percent of vote by party - My Step Alliance 70.4%, BHK 8.3%, Bright Armenia 6.4%, RPA 4.7%, ARF 3.9%, other 6.3%; seats by party - My Step Alliance 88, BHK 26, Bright Armenia 18; composition - men 112, women 20, percent of women 15.2%"
},
"judicial_branch": {
"highest_courts": "Court of Cassation (consists of the Criminal Chamber with a chairman and 5 judges and the Civil and Administrative Chamber with a chairman and 10 judges – with both civil and administrative specializations); Constitutional Court (consists of 9 judges)",
"judge_selection_and_term_of_office": "Court of Cassation judges nominated by the Supreme Judicial Council, a 10-member body of selected judges and legal scholars; judges appointed by the president; judges can serve until age 65; Constitutional Court judges - 4 appointed by the president, and 5 elected by the National Assembly; judges can serve until age 70",
"subordinate_courts": "criminal and civil appellate courts; administrative appellate court; first instance courts; specialized administrative and bankruptcy courts"
},
"political_parties_and_leaders": {
"parties": [
{
"name": "Armenian National Congress",
"name_alternative": "ANC",
"leaders": [
"Levon TER-PETROSSIAN"
],
"note": "bloc of independent and opposition parties"
},
{
"name": "Armenian National Movement",
"name_alternative": "ANM",
"leaders": [
"Ararat ZURABIAN"
]
},
{
"name": "Armenian Revolutionary Federation",
"name_alternative": "ARF",
"leaders": [
"Hrant MARKARIAN"
],
"note": "\"Dashnak\" Party"
},
{
"name": "Bright Armenia",
"leaders": [
"Edmon MARUKYAN"
]
},
{
"name": "Christian Democratic Rebirth Party",
"leaders": [
"Levon SHIRINYAN"
]
},
{
"name": "Citizen's Decision",
"leaders": [
"Suren SAHAKYAN"
]
},
{
"name": "Civil Contract",
"leaders": [
"Nikol PASHINYAN"
]
},
{
"name": "Heritage Party",
"leaders": [
"Raffi HOVHANNISIAN"
]
},
{
"name": "Mission Party",
"leaders": [
"Manuk SUKIASYAN"
]
},
{
"name": "My Step Alliance ",
"leaders": [
"Nikol PASHINYAN"
],
"note": "Civil Contract Party and Mission Party"
},
{
"name": "National Progress Party",
"leaders": [
"Lusine HAROYAN"
]
},
{
"name": "People's Party of Armenia",
"leaders": [
"Stepan DEMIRCHIAN"
]
},
{
"name": "Prosperous Armenia",
"name_alternative": "BHK",
"leaders": [
"Gagik TSARUKYAN"
]
},
{
"name": "Republic",
"leaders": [
"Aram SARGSYAN"
]
},
{
"name": "Republican Party of Armenia",
"name_alternative": "RPA",
"leaders": [
"Serzh SARGSIAN"
]
},
{
"name": "Rule of Law Party",
"name_alternative": "OEK",
"leaders": [
"Artur BAGHDASARIAN"
],
"note": "Orinats Yerkir"
},
{
"name": "Sasna Tser",
"leaders": [
"Varuzhan AVETISYAN"
]
},
{
"name": "We Alliance",
"name_alternative": "FD-H",
"leaders": [
"Aram SARGSIAN"
]
}
]
},
"international_organization_participation": [
{
"organization": "ADB"
},
{
"organization": "BSEC"
},
{
"organization": "CD"
},
{
"organization": "CE"
},
{
"organization": "CIS"
},
{
"organization": "CSTO"
},
{
"organization": "EAEC ",
"note": "observer"
},
{
"organization": "EAEU"
},
{
"organization": "EAPC"
},
{
"organization": "EBRD"
},
{
"organization": "FAO"
},
{
"organization": "GCTU"
},
{
"organization": "IAEA"
},
{
"organization": "IBRD"
},
{
"organization": "ICAO"
},
{
"organization": "ICC ",
"note": "NGOs"
},
{
"organization": "ICRM"
},
{
"organization": "IDA"
},
{
"organization": "IFAD"
},
{
"organization": "IFC"
},
{
"organization": "IFRCS"
},
{
"organization": "ILO"
},
{
"organization": "IMF"
},
{
"organization": "Interpol"
},
{
"organization": "IOC"
},
{
"organization": "IOM"
},
{
"organization": "IPU"
},
{
"organization": "ISO"
},
{
"organization": "ITSO"
},
{
"organization": "ITU"
},
{
"organization": "MIGA"
},
{
"organization": "NAM ",
"note": "observer"
},
{
"organization": "OAS ",
"note": "observer"
},
{
"organization": "OIF"
},
{
"organization": "OPCW"
},
{
"organization": "OSCE"
},
{
"organization": "PFP"
},
{
"organization": "UN"
},
{
"organization": "UNCTAD"
},
{
"organization": "UNESCO"
},
{
"organization": "UNIDO"
},
{
"organization": "UNIFIL"
},
{
"organization": "UNWTO"
},
{
"organization": "UPU"
},
{
"organization": "WCO"
},
{
"organization": "WFTU ",
"note": "NGOs"
},
{
"organization": "WHO"
},
{
"organization": "WIPO"
},
{
"organization": "WMO"
},
{
"organization": "WTO"
}
],
"diplomatic_representation": {
"in_united_states": {
"chief_of_mission": "Ambassador Varuzhan NERSESSYAN (since 11 January 2019)",
"chancery": "2225 R Street NW, Washington, DC 20008",
"telephone": "[1] (202) 319-1976",
"fax": "[1] (202) 319-2982",
"consulates_general": "Glendale (CA)"
},
"from_united_states": {
"chief_of_mission": "Ambassador Richard MILLS (since 13 February 2015)",
"embassy": "1 American Ave., Yerevan 0082",
"mailing_address": "American Embassy Yerevan, US Department of State, 7020 Yerevan Place, Washington, DC 20521-7020",
"telephone": "[374](10) 464-700",
"fax": "[374](10) 464-742"
}
},
"flag_description": {
"description": "three equal horizontal bands of red (top), blue, and orange; the color red recalls the blood shed for liberty, blue the Armenian skies as well as hope, and orange the land and the courage of the workers who farm it"
},
"national_symbol": {
"symbols": [
{
"symbol": "Mount Ararat"
},
{
"symbol": "eagle"
},
{
"symbol": "lion"
}
],
"colors": [
{
"color": "red"
},
{
"color": "blue"
},
{
"color": "orange"
}
]
},
"national_anthem": {
"name": "\"Mer Hayrenik\" (Our Fatherland)",
"lyrics_music": "Mikael NALBANDIAN/Barsegh KANACHYAN",
"note": "adopted 1991; based on the anthem of the Democratic Republic of Armenia (1918-1922) but with different lyrics",
"audio_url": "https://www.cia.gov/library/publications/the-world-factbook/attachments/audios/original/AM.mp3?1538604749"
}
},
"economy": {
"overview": "Under the old Soviet central planning system, Armenia developed a modern industrial sector, supplying machine tools, textiles, and other manufactured goods to sister republics, in exchange for raw materials and energy. Armenia has since switched to small-scale agriculture and away from the large agro industrial complexes of the Soviet era. Armenia has only two open trade borders - Iran and Georgia - because its borders with Azerbaijan and Turkey have been closed since 1991 and 1993, respectively, as a result of Armenia's ongoing conflict with Azerbaijan over the separatist Nagorno-Karabakh region.\nArmenia joined the World Trade Organization in January 2003. The government has made some improvements in tax and customs administration in recent years, but anti-corruption measures have been largely ineffective. Armenia will need to pursue additional economic reforms and strengthen the rule of law in order to raise its economic growth and improve economic competitiveness and employment opportunities, especially given its economic isolation from Turkey and Azerbaijan.\nArmenia's geographic isolation, a narrow export base, and pervasive monopolies in important business sectors have made it particularly vulnerable to volatility in the global commodity markets and the economic challenges in Russia. Armenia is particularly dependent on Russian commercial and governmental support, as most key Armenian infrastructure is Russian-owned and/or managed, especially in the energy sector. Remittances from expatriates working in Russia are equivalent to about 12-14% of GDP. Armenia joined the Russia-led Eurasian Economic Union in January 2015, but has remained interested in pursuing closer ties with the EU as well, signing a Comprehensive and Enhanced Partnership Agreement with the EU in November 2017. Armenia’s rising government debt is leading Yerevan to tighten its fiscal policies – the amount is approaching the debt to GDP ratio threshold set by national legislation.",
"gdp": {
"purchasing_power_parity": {
"annual_values": [
{
"value": 28340000000,
"units": "USD",
"date": "2017"
},
{
"value": 26370000000,
"units": "USD",
"date": "2016"
},
{
"value": 26300000000,
"units": "USD",
"date": "2015"
}
],
"global_rank": 136,
"note": "data are in 2017 dollars"
},
"official_exchange_rate": {
"USD": 11540000000,
"date": "2017"
},
"real_growth_rate": {
"annual_values": [
{
"value": 7.5,
"units": "%",
"date": "2017"
},
{
"value": 0.3,
"units": "%",
"date": "2016"
},
{
"value": 3.3,
"units": "%",
"date": "2015"
}
],
"global_rank": 12
},
"per_capita_purchasing_power_parity": {
"annual_values": [
{
"value": 9500,
"units": "USD",
"date": "2017"
},
{
"value": 8800,
"units": "USD",
"date": "2016"
},
{
"value": 8800,
"units": "USD",
"date": "2015"
}
],
"global_rank": 142,
"note": "data are in 2017 dollars"
},
"composition": {
"by_end_use": {
"end_uses": {
"household_consumption": {
"value": 76.7,
"units": "%"
},
"government_consumption": {
"value": 14.2,
"units": "%"
},
"investment_in_fixed_capital": {
"value": 17.3,
"units": "%"
},
"investment_in_inventories": {
"value": 4.1,
"units": "%"
},
"exports_of_goods_and_services": {
"value": 38.1,
"units": "%"
},
"imports_of_goods_and_services": {
"value": -50.4,
"units": "%"
}
},
"date": "2017"
},
"by_sector_of_origin": {
"sectors": {
"agriculture": {
"value": 16.7,
"units": "%"
},
"industry": {
"value": 28.2,
"units": "%"
},
"services": {
"value": 54.8,
"units": "%"
}
},
"date": "2017"
}
}
},
"gross_national_saving": {
"annual_values": [
{
"value": 17.8,
"units": "percent_of_gdp",
"date": "2017"
},
{
"value": 16.6,
"units": "percent_of_gdp",
"date": "2016"
},
{
"value": 18.4,
"units": "percent_of_gdp",
"date": "2015"
}
],
"global_rank": 112
},
"agriculture_products": {
"products": [
"fruit",
"grapes",
"apricots",
"vegetables",
"livestock"
]
},
"industries": {
"industries": [
"brandy",
"mining",
"diamond processing",
"metal-cutting machine tools",
"forging",
"pressing machines",
"electric motors",
"knitted wear",
"hosiery",
"shoes",
"silk fabric",
"chemicals",
"trucks",
"instruments",
"microelectronics",
"jewelry",
"software",
"food processing"
]
},
"industrial_production_growth_rate": {
"annual_percentage_increase": 5.4,
"global_rank": 51,
"date": "2017"
},
"labor_force": {
"total_size": {
"total_people": 1507000,
"global_rank": 131,
"date": "2017"
},
"by_occupation": {
"occupation": {
"agriculture": {
"value": 36.3,
"units": "%"
},
"industry": {
"value": 17,
"units": "%"
},
"services": {
"value": 46.7,
"units": "%"
}
},
"date": "2013"
}
},
"unemployment_rate": {
"annual_values": [
{
"value": 18.9,
"units": "%",
"date": "2017"
},
{
"value": 18.8,
"units": "%",
"date": "2016"
}
],
"global_rank": 183
},
"population_below_poverty_line": {
"value": 32,
"units": "%",
"date": "2013"
},
"household_income_by_percentage_share": {
"lowest_ten_percent": {
"value": 3.5,
"units": "%"
},
"highest_ten_percent": {
"value": 25.7,
"units": "%"
},
"date": "2014"
},
"distribution_of_family_income": {
"annual_values": [
{
"value": 31.5,
"units": "gini_index",
"date": "2014"
},
{
"value": 31.5,
"units": "gini_index",
"date": "2013"
}
],
"global_rank": 125
},
"budget": {
"revenues": {
"value": 2644000000,
"units": "USD"
},
"expenditures": {
"value": 3192000000,
"units": "USD"
},
"date": "2017"
},
"taxes_and_other_revenues": {
"percent_of_gdp": 22.9,
"global_rank": 130,
"date": "2017"
},
"budget_surplus_or_deficit": {
"percent_of_gdp": -4.8,
"global_rank": 167,
"date": "2017"
},
"public_debt": {
"annual_values": [
{
"value": 53.5,
"units": "percent_of_gdp",
"date": "2017"
},
{
"value": 51.9,
"units": "percent_of_gdp",
"date": "2016"
}
],
"global_rank": 89
},
"fiscal_year": {
"start": "1 January",
"end": "31 December"
},
"inflation_rate": {
"annual_values": [
{
"value": 0.9,
"units": "%",
"date": "2017"
},
{
"value": -1.4,
"units": "%",
"date": "2016"
}
],
"global_rank": 44
},
"central_bank_discount_rate": {
"annual_values": [
{
"value": 6.5,
"units": "%",
"date": "2016-12-14"
},
{
"value": 10.5,
"units": "%",
"date": "2015-02-10"
}
],
"global_rank": 57,
"note": "this is the Refinancing Rate, the key monetary policy instrument of the Armenian National Bank"
},
"commercial_bank_prime_lending_rate": {
"annual_values": [
{
"value": 14.41,
"units": "%",
"date": "2017-12-31"
},
{
"value": 17.36,
"units": "%",
"date": "2016-12-31"
}
],
"global_rank": 47,
"note": "average lending rate on loans up to one year"
},
"stock_of_narrow_money": {
"annual_values": [
{
"value": 1629000000,
"units": "USD",
"date": "2017-12-31"
},
{
"value": 1355000000,
"units": "USD",
"date": "2016-12-31"
}
],
"global_rank": 141
},
"stock_of_broad_money": {
"annual_values": [
{
"value": 1629000000,
"units": "USD",
"date": "2017-12-31"
},
{
"value": 1355000000,
"units": "USD",
"date": "2016-12-31"
}
],
"global_rank": 149
},
"stock_of_domestic_credit": {
"annual_values": [
{
"value": 6712000000,
"units": "USD",
"date": "2017-12-31"
},
{
"value": 5689000000,
"units": "USD",
"date": "2016-12-31"
}
],
"global_rank": 120
},
"market_value_of_publicly_traded_shares": {
"annual_values": [
{
"value": 132100000,
"units": "USD",
"date": "2012-12-31"
},
{
"value": 139600000,
"units": "USD",
"date": "2011-12-31"
},
{
"value": 144800000,
"units": "USD",
"date": "2010-12-31"
}
],
"global_rank": 121
},
"current_account_balance": {
"annual_values": [
{
"value": -328000000,
"units": "USD",
"date": "2017"
},
{
"value": -238000000,
"units": "USD",
"date": "2016"
}
],
"global_rank": 107
},
"exports": {
"total_value": {
"annual_values": [
{
"value": 2361000000,
"units": "USD",
"date": "2017"
},
{
"value": 1891000000,
"units": "USD",
"date": "2016"
}
],
"global_rank": 135
},
"commodities": {
"by_commodity": [
"unwrought copper",
"pig iron",
"nonferrous metals",
"gold",
"diamonds",
"mineral products",
"foodstuffs",
"brandy",
"cigarettes",
"energy"
]
},
"partners": {
"by_country": [
{
"name": "Russia",
"percent": 24.2
},
{
"name": "Bulgaria",
"percent": 12.8
},
{
"name": "Switzerland",
"percent": 12
},
{
"name": "Georgia",
"percent": 6.9
},
{
"name": "Germany",
"percent": 5.9
},
{
"name": "China",
"percent": 5.5
},
{
"name": "Iraq",
"percent": 5.4
},
{
"name": "UAE",
"percent": 4.6
},
{
"name": "Netherlands",
"percent": 4.1
}
],
"date": "2017"
}
},
"imports": {
"total_value": {
"annual_values": [
{
"value": 3771000000,
"units": "USD",
"date": "2017"
},
{
"value": 2835000000,
"units": "USD",
"date": "2016"
}
],
"global_rank": 142
},
"commodities": {
"by_commodity": [
"natural gas",
"petroleum",
"tobacco products",
"foodstuffs",
"diamonds",
"pharmaceuticals",
"cars"
]
},
"partners": {
"by_country": [
{
"name": "Russia",
"percent": 28
},
{
"name": "China",
"percent": 11.5
},
{
"name": "Turkey",
"percent": 5.5
},
{
"name": "Germany",
"percent": 4.9
},
{
"name": "Iran",
"percent": 4.3
}
],
"date": "2017"
}
},
"reserves_of_foreign_exchange_and_gold": {
"annual_values": [
{
"value": 2314000000,
"units": "USD",
"date": "2017-12-31"
},
{
"value": 2204000000,
"units": "USD",
"date": "2016-12-31"
}
],
"global_rank": 119
},
"external_debt": {
"annual_values": [
{
"value": 10410000000,
"units": "USD",
"date": "2017-12-31"
},
{
"value": 8987000000,
"units": "USD",
"date": "2016-12-31"
}
],
"global_rank": 113
},
"stock_of_direct_foreign_investment": {
"at_home": {
"annual_values": [
{
"value": 4168999999.9999995,
"units": "USD",
"date": "2015"
},
{
"value": 4086999999.9999995,
"units": "USD",
"date": "2014-12-31"
}
],
"global_rank": 109
},
"abroad": {
"annual_values": [
{
"value": 228000000,
"units": "USD",
"date": "2015"
},
{
"value": 215000000,
"units": "USD",
"date": "2014"
}
],
"global_rank": 107
}
},
"exchange_rates": {
"annual_values": [
{
"value": 487.9,
"units": "USD",
"date": "2017"
},
{
"value": 480.49,
"units": "USD",
"date": "2016"
},
{
"value": 480.49,
"units": "USD",
"date": "2015"
},
{
"value": 477.92,
"units": "USD",
"date": "2014"
},
{
"value": 415.92,
"units": "USD",
"date": "2013"
}
],
"note": "drams (AMD) per US dollar"
}
},
"energy": {
"electricity": {
"access": {
"total_electrification": {
"value": 100,
"units": "%"
},
"date": "2016"
},
"production": {
"kWh": 6951000000,
"global_rank": 112,
"date": "2016"
},
"consumption": {
"kWh": 5291000000,
"global_rank": 121,
"date": "2016"
},
"exports": {
"kWh": 1424000000,
"global_rank": 50,
"date": "2015"
},
"imports": {
"kWh": 275000000,
"global_rank": 90,
"date": "2016"
},
"installed_generating_capacity": {
"kW": 4080000,
"global_rank": 86,
"date": "2016"
},
"by_source": {
"fossil_fuels": {
"percent": 58,
"global_rank": 134,
"date": "2016"
},
"nuclear_fuels": {
"percent": 9,
"global_rank": 15,
"date": "2017"
},
"hydroelectric_plants": {
"percent": 32,
"global_rank": 65,
"date": "2017"
},
"other_renewable_sources": {
"percent": 0,
"global_rank": 173,
"date": "2017"
}
}
},
"crude_oil": {
"production": {
"bbl_per_day": 0,
"global_rank": 105,
"date": "2017"
},
"exports": {
"bbl_per_day": 0,
"global_rank": 86,
"date": "2015"
},
"imports": {
"bbl_per_day": 0,
"global_rank": 90,
"date": "2015"
},
"proved_reserves": {
"bbl": 0,
"global_rank": 103,
"date": "2018-01-01"
}
},
"refined_petroleum_products": {
"production": {
"bbl_per_day": 0,
"global_rank": 114,
"date": "2015"
},
"consumption": {
"bbl_per_day": 8000,
"global_rank": 162,
"date": "2016"
},
"exports": {
"bbl_per_day": 0,
"global_rank": 126,
"date": "2015"
},
"imports": {
"bbl_per_day": 7145,
"global_rank": 158,
"date": "2015"
}
},
"natural_gas": {
"production": {
"cubic_metres": 0,
"global_rank": 100,
"date": "2017"
},
"consumption": {
"cubic_metres": 2350000000,
"global_rank": 80,
"date": "2017"
},
"exports": {
"cubic_metres": 0,
"global_rank": 62,
"date": "2017"
},
"imports": {
"cubic_metres": 2350000000,
"global_rank": 48,
"date": "2017"
},
"proved_reserves": {
"cubic_metres": 0,
"global_rank": 106,
"date": "2014-01-01"
}
},
"carbon_dioxide_emissions_from_consumption_of_energy": {
"megatonnes": 5501000,
"global_rank": 131,
"date": "2017"
}
},
"communications": {
"telephones": {
"fixed_lines": {
"total_subscriptions": 505190,
"subscriptions_per_one_hundred_inhabitants": 17,
"global_rank": 94,
"date": "2017"
},
"mobile_cellular": {
"total_subscriptions": 3488524,
"subscriptions_per_one_hundred_inhabitants": 115,
"global_rank": 135,
"date": "2017"
},
"system": {
"general_assessment": "telecommunications investments have made major inroads in modernizing and upgrading the outdated telecommunications network inherited from the Soviet era; now 100% privately owned and undergoing modernization and expansion; with a small populaton and low GDP - moderate growth in mobile market; mobile operators promise mobile broadband to be faster (2017)",
"domestic": "17 per 100 fixed-line, 115 per 100 mobile-cellular; reliable fixed-line and mobile-cellular services are available across Yerevan and in major cities and towns; mobile-cellular coverage available in most rural areas (2017)",
"international": "country code - 374; Yerevan is connected to the Trans-Asia-Europe fiber-optic cable through Iran; additional international service is available by microwave radio relay and landline connections to the other countries of the Commonwealth of Independent States, through the Moscow international switch, and by satellite to the rest of the world; satellite earth stations - 3 (2015)"
}
},
"broadcast_media": "2 public TV networks operating alongside about 40 privately owned TV stations that provide local to near nationwide coverage; major Russian broadcast stations are widely available; subscription cable TV services are available in most regions; Armenian TV completed conversion from analog to digital broadcasting in late 2016; Public Radio of Armenia is a national, state-run broadcast network that operates alongside 21 privately owned radio stations; several major international broadcasters are available (2017)",
"internet": {
"country_code": ".am",
"users": {
"total": 1891775,
"percent_of_population": 62,
"global_rank": 116,
"date": "2016-07-01"
}
}
},
"transportation": {
"air_transport": {
"national_system": {
"number_of_registered_air_carriers": 3,
"inventory_of_registered_aircraft_operated_by_air_carriers": 5,
"date": "2015"
},
"civil_aircraft_registration_country_code_prefix": {
"prefix": "EK",
"date": "2016"
},
"airports": {
"total": {
"airports": 11,
"global_rank": 153,
"date": "2013"
},
"paved": {
"total": 10,
"over_3047_metres": 2,
"2438_to_3047_metres": 2,
"1524_to_2437_metres": 4,
"914_to_1523_metres": 2,
"date": "2017"
},
"unpaved": {
"total": 1,
"914_to_1523_metres": 1,
"date": "2013"
}
}
},
"pipelines": {
"by_type": [
{
"type": "gas (high and medium pressure)",
"length": 3838,
"units": "km"
}
],
"date": "2017"
},
"railways": {
"total": {
"length": 780,
"units": "km"
},
"broad_gauge": {
"length": 780,
"electrified": 780,
"units": "km\n1.520-m",
"gauge": "gauge"
},
"note": {
"length": 726,
"units": "km"
},
"global_rank": 98,
"date": "2014"
},
"roadways": {
"total": {
"value": 7792,
"units": "km"
},
"global_rank": 95,
"date": "2013"
}
},
"military_and_security": {
"expenditures": {
"annual_values": [
{
"value": 4.2,
"units": "percent_of_gdp",
"date": "2018"
},
{
"value": 4,
"units": "percent_of_gdp",
"date": "2017"
},
{
"value": 4.09,
"units": "percent_of_gdp",
"date": "2016"
},
{
"value": 4.25,
"units": "percent_of_gdp",
"date": "2015"
},
{
"value": 3.94,
"units": "percent_of_gdp",
"date": "2014"
}
],
"global_rank": 13
},
"branches": {
"by_name": [
"Armenian Armed Forces",
"Ground Forces",
"Air Force",
"Air Defense",
"\"Nagorno-Karabakh Republic\"",
"Nagorno-Karabakh Self-Defense Force (NKSDF)"
],
"date": "2011"
},
"service_age_and_obligation": {
"years_of_age": 18,
"note": "18-27 years of age for voluntary or compulsory military service; 2-year conscript service obligation; 17 year olds are eligible to become cadets at military higher education institutes, where they are classified as military personnel",
"date": "2012"
}
},
"transnational_issues": {
"disputes": [
"the dispute over the break-away Nagorno-Karabakh region and the Armenian military occupation of surrounding lands in Azerbaijan remains the primary focus of regional instability",
"residents have evacuated the former Soviet-era small ethnic enclaves in Armenia and Azerbaijan",
"Turkish authorities have complained that blasting from quarries in Armenia might be damaging the medieval ruins of Ani, on the other side of the Arpacay valley",
"in 2009, Swiss mediators facilitated an accord reestablishing diplomatic ties between Armenia and Turkey, but neither side has ratified the agreement and the rapprochement effort has faltered",
"local border forces struggle to control the illegal transit of goods and people across the porous, undemarcated Armenian, Azerbaijani, and Georgian borders",
"ethnic Armenian groups in the Javakheti region of Georgia seek greater autonomy from the Georgian Government"
],
"refugees_and_iternally_displaced_persons": {
"refugees": {
"by_country": [
{
"people": 14680,
"country_of_origin": "Syria - ethnic Armenians"
}
],
"date": "2017"
},
"stateless_persons": {
"people": 773,
"date": "2017"
}
},
"illicit_drugs": {
"note": "illicit cultivation of small amount of cannabis for domestic consumption; minor transit point for illicit drugs - mostly opium and hashish - moving from Southwest Asia to Russia and to a lesser extent the rest of Europe"
}
}
},
"metadata": {
"date": "2019-01-23",
"source": "https://web.archive.org/web/20190123225802/https://www.cia.gov/library/publications/the-world-factbook/geos/am.html",
"nearby_dates": "https://web.archive.org/web/20190123000000*/https://www.cia.gov/library/publications/the-world-factbook/geos/am.html"
}
}
JSON Country record for Armenia
As you can see, the data is extensive. How should we store this data in Microsoft Azure? One answer is to extract the JSON for each country and store it in blob storage. With this we could create a web page for displaying country data; after a user selects a country, code in the page would grab the country JSON with a single web request and access it as a variable. This indeed is one way we'll be storing data, in Azure blob storage.
In addition to the JSON data, there is a flag image and map image available on the World Factbook site in the form of .gif files. If we retrieve those, blob storage would be the logical place to store them.
Flag Image for Armenia
Map Image for Armenia
While useful, blob storage alone isn't enough. We need more in order to accomplish the following:
While useful, blob storage alone isn't enough. We need more in order to accomplish the following:
- Seach the data
- Report on the data
- Analyze the data
- Chart the data
We need to have the data in a database for the above activities. A relational database wouldn't make sense here, but a document database is ready-made for this kind of JSON data. So, we'll also be storing our country JSON in a Cosmos DB database. Once this data is in Cosmos DB, we'll be able to mine it richly.
Do we really need to store the country JSON data in two places (blob and Cosmos DB)? No, but depending on your needs you might opt for blob storage or Cosmos DB; so we'll do both just to show how.
Do we really need to store the country JSON data in two places (blob and Cosmos DB)? No, but depending on your needs you might opt for blob storage or Cosmos DB; so we'll do both just to show how.
Country Names and Keys
Before we go any further we need to discuss how countries are referenced and organized. The World Factbook JSON data is organized into countries (as well as pseudo-countries and regions). Every country has a country name, such as "Argentina" or "Falkland Islands (Islas Malvinas)".If we take a look at the Factbook JSON, however, we see country records are organized by country key, not country name. A country key is a derivative of the country name, but is lower case, replaces spaces and hyphens with underscores, and drops some special characters like commas and parentheses.
Factbook JSON is Organized by Country Key
Following these rules, here are some examples:
- The country key for "Argentina" is "argentina".
- The country key for "Falkland Islands (Islas Malvinas)" is "falkland_islands_islas_malvinas".
- The country key for "Micronesia, Federated States Of" is "micronesia_federated_states_of".
There's one other country identifier we need, a 2-letter code such as "AL" for Albania or "AU" for Austria. The flag and map images we'll be retrieving use 2-letter country codes in their filenames.
We'll need the exact list of country codes, country names, and country keys in order to successfully locate the country records and image files. In our code, the Country class holds that data.
Blob Storage
In our worldfactbook storage account, we create a container named data. We will store 3 kinds of blobs in this container:- JSON country record
country-key.json - example: falkland_islands_islas_malvinas.json - flag image
country-key.gif - example: falkland_islands_islas_malvinas.gif - map image
country-key-map.gif - example: falkland_islands_islas_malvinas-map.gif
Database
We'll use Cosmos DB and create a Country collection, with one document per country.
Country Collection
We create a database named Factbook, and within it a collection named Country. We'll be using the property /source for partition key which will have the same value "Factbook" for all records, meaning they'll all occupy the same partition. Although the JSON documents have a lot of detail, there only 260 of them and the total data size is well under the 10GB limit for a partition. The Id for a collection document is /name, its country name.
Defining Country Collection
Azure Durable Function to Retrieve Data
Now that we've determined how we'll be storing data, we need to write the code to retrieve the data and store it. An Azure Function with a Timer Trigger is the ideal vehicle for this: the JSON data is updated once a week, so we'd like our data retrieval/update code to run once a week. A Timer Trigger lets us do just that, schedule how often the function runs.
However, a regular Azure Function won't quite fit our needs. It will take some time to download this data and store various parts of it in Cosmos DB and blob storage. A standard Azure Function has a maximum execution time of 10 minutes, and we can't be sure we'll be done in 10 minutes. Fortunately, Azure Durable Functions exist. Durable functions allow us to chain a number of function calls, and they support patterns for parallel processing. They also maintain state, checkpoints, and handle restarts: in other words, they can be counted on to get the job done even if the function gets interrupted.
Azure Durable Function: Update
Our durable function will be named Update, and we'll be writing this in Visual Studio in C# / .NET Core. Working locally in Visual Studio has a number of nice benefits, including project/function templates and the ability to run and test locally.
We'll be looking at areas of the code section by section below, but if you want to see all of it visit https://github.com/davidpallmann/world-factbook-indexer.
We'll be looking at areas of the code section by section below, but if you want to see all of it visit https://github.com/davidpallmann/world-factbook-indexer.
Configuration
At the top of Update.cs, our function code, is some configuration information and global variables.
namespace WorldFactbookIndexer
{
public static class Update
{
// Data source URLs
const String DownloadUrl = "https://github.com/iancoleman/cia_world_factbook_api/raw/master/data/factbook.json";
const String FlagUrlFormat = "https://www.cia.gov/library/publications/resources/the-world-factbook/attachments/flags/{0}-flag.gif";
const String MapUrlFormat = "https://www.cia.gov/library/publications/resources/the-world-factbook/attachments/maps/{0}-map.gif";
// Storage account - set to your storage account.
const String StorageName = "your-storage-account-name";
const String StorageKey = "...your-storage-access-key";
const String ContainerName = "data";
// Cosmos DB
private const string EndpointUrl = "https://your-cosmo-db.documents.azure.com:443/";
private const string PrimaryKey = "...your-cosmo-db-access-key...";
private static DocumentClient client = new DocumentClient(new Uri(EndpointUrl), PrimaryKey);
private static Uri CountryUrl = UriFactory.CreateDocumentCollectionUri("Factbook", "Country");
// Global variables
private static bool LocalTest = false; // If true, we are running locally. If false, we're running in the cloud.
private static JObject data = null; // JSON country data
private static Country[] countries = Country.List(); // Master country list
Top of Update.cs
These items will be referenced in the code we'll be looking at, so let's get a quick explanation:
These items will be referenced in the code we'll be looking at, so let's get a quick explanation:
- DownloadUrl is the URL for the JSON file we'll be retrieving once a week from Ian Coleman's github.
- FlagUrlFormat and MapUrlFormat are String.Format strings for image URLs on the World Factbook site.
- StorageName and StorageKey are credentials for the Azure storage account.
- ContainerName is the blob container name.
- EndpointUrl and PrimaryKey are the URL and key needed for accessing our Cosmos DB database.
- DocumentClient is the document client we'll use to access the database.
- CountryUrl is the Country collection URL.
- LocalTest is a flag set to true when testing locally (when true, we can assume local files persist).
- data is a JObject that will hold the bulk JSON after it is downloaded.
- countries is our country list.
Triggers
A start function connects a trigger event to our durable function. During development and debugging, I configured the start function for Update with an HTTP trigger. This allowed me to run the function locally, and simply hit a URL in a browser to start it running.// Start function for HTTP trigger - used during initial development & running locally to debug
[FunctionName("Update_HttpStart")]
public static async Task<HttpResponseMessage> HttpStart(
[HttpTrigger(AuthorizationLevel.Anonymous, "get", "post")]HttpRequestMessage req,
[OrchestrationClient]DurableOrchestrationClient starter,
ILogger log)
{
// Function input comes from the request content.
string instanceId = await starter.StartNewAsync("Update", null);
log.LogInformation($"Started orchestration with ID = '{instanceId}'.");
return starter.CreateCheckStatusResponse(req, instanceId);
}
HTTP Trigger Start Function, Used for Local Development and Testing
When running the function out of Visual Studio, a console window appears and displays the trigger URL. Entering this URL in a browser kicks off a run of the function.
Function Running Locally with HTTP Trigger
When development and testing was complete, the trigger was changed to a Timer trigger scheduled to run once a week on Saturday.
// Start function for timer trigger
// CRON expressions cheat sheet: https://codehollow.com/2017/02/azure-functions-time-trigger-cron-cheat-sheet/
const String EverySaturday = "0 0 0 * * 5"; // every Saturday
const String EveryHour = "0 2 * * * *"; // every hour on the hour
[FunctionName("Update_TimerStart")]
public static async Task Update_TimerStart([TimerTrigger(EverySaturday)]TimerInfo timerInfo,
[OrchestrationClient] DurableOrchestrationClient starter, ILogger log)
{
string instanceId = await starter.StartNewAsync("Update", null);
log.LogInformation($"================ Started orchestration with ID = '{instanceId}'. at " + DateTime.Now.ToString());
}
Timer Trigger Start Function, Keeps Data Up-to-Date
Once published to our Azure Function (after creating the function in the Azure Portal and downloading a Publish Profile), the function will run on its own once a week.
The Update Function and the Fan-Out/Fan-In Pattern
When a trigger fires, the Update durable function starts running. Let's take a look at the code, then we'll explain what it's doing.// Update : Azure Durable Function for loading/updating Azure storage and database with World Factbook data
[FunctionName("Update")]
public static async Task<List<bool>> RunOrchestrator([OrchestrationTrigger] DurableOrchestrationContext context)
{
DurableOrchestrationContext Context = context;
var outputs = new List<bool>();
// Download bulk dataset for all countries
bool haveData = await context.CallActivityAsync<bool>("DownloadData", "unused");
outputs.Add(haveData);
if (haveData)
{
// Use fan-out / fan-in to store a dataset for each country
var tasks = new Task<bool>[countries.Length];
for (int i = 0; i < countries.Length; i++)
{
tasks[i] = context.CallActivityAsync<bool>("UploadCountryData", countries[i]);
}
await Task.WhenAll(tasks);
foreach(Task<bool> t in tasks)
{
outputs.Add(t.Result);
}
outputs.Add(true);
}
return outputs;
}
Update Function
Note the way the function is invoked, with context.CallActivityAsync. In a durable function, other functions must be called this way in order to preserve state and create recovery checkpoints.
Country Class
The Country class represents a country, and contains three properties: Code, Name, and Key. Key is computed from Name automatically. Country also has a static method List, which returns the full country list.using System;
using System.Collections.Generic;
using System.Text;
namespace WorldFactbookIndexer
{
public class Country
{
public String Code { get; set; }
public String Name { get; set; }
// Return key that matches JSON key in list of country objects.
// Calculated as lower-case edition of name, with commas+parentheses removed and spaces+hyphens replaced with underscores
public String Key
{
get
{
if (String.IsNullOrEmpty(Name))
return this.Name;
else
return this.Name.Replace(",","").Replace("(","").Replace(")","").Replace("-","_").Replace(" ", "_").ToLower();
}
}
public Country() { }
public Country(String code, String name)
{
this.Code = code;
this.Name = name;
}
// List - return country list.
public static Country[] List()
{
return new Country[]
{
new Country("xx", "World"),
new Country("af", "Afghanistan"),
new Country("ax", "Akrotiri"),
new Country("al", "Albania"),
new Country("ag", "Algeria"),
new Country("aq", "American Samoa"),
new Country("an", "Andorra"),
new Country("ao", "Angola"),
new Country("av", "Anguilla"),
new Country("ay", "Antarctica"),
new Country("ac", "Antigua And Barbuda"),
new Country("xq", "Arctic Ocean"),
new Country("ar", "Argentina"),
new Country("am", "Armenia"),
new Country("aa", "Aruba"),
new Country("at", "Ashmore And Cartier Islands"),
new Country("zh", "Atlantic Ocean"),
new Country("as", "Australia"),
new Country("au", "Austria"),
new Country("aj", "Azerbaijan"),
new Country("bf", "Bahamas, The"),
new Country("ba", "Bahrain"),
new Country("bg", "Bangladesh"),
new Country("bb", "Barbados"),
new Country("bo", "Belarus"),
new Country("be", "Belgium"),
new Country("bh", "Belize"),
new Country("bn", "Benin"),
new Country("bd", "Bermuda"),
new Country("bt", "Bhutan"),
new Country("bl", "Bolivia"),
new Country("bk", "Bosnia And Herzegovina"),
new Country("bc", "Botswana"),
new Country("bv", "Bouvet Island"),
new Country("br", "Brazil"),
new Country("io", "British Indian Ocean Territory"),
new Country("vi", "British Virgin Islands"),
new Country("bx", "Brunei"),
new Country("bu", "Bulgaria"),
new Country("uv", "Burkina Faso"),
new Country("bm", "Burma"),
new Country("by", "Burundi"),
new Country("cv", "Cabo Verde"),
new Country("cb", "Cambodia"),
new Country("cm", "Cameroon"),
new Country("ca", "Canada"),
new Country("cj", "Cayman Islands"),
new Country("ct", "Central African Republic"),
new Country("cd", "Chad"),
new Country("ci", "Chile"),
new Country("ch", "China"),
new Country("kt", "Christmas Island"),
new Country("ip", "Clipperton Island"),
new Country("ck", "Cocos (Keeling) Islands"),
new Country("co", "Colombia"),
new Country("cn", "Comoros"),
new Country("cg", "Congo, Democratic Republic Of The"),
new Country("cf", "Congo, Republic Of The"),
new Country("cw", "Cook Islands"),
new Country("cr", "Coral Sea Islands"),
new Country("cs", "Costa Rica"),
new Country("iv", "Cote D'Ivoire"),
new Country("hr", "Croatia"),
new Country("cu", "Cuba"),
new Country("uc", "Curacao"),
new Country("cy", "Cyprus"),
new Country("ez", "Czechia"),
new Country("da", "Denmark"),
new Country("dx", "Dhekelia"),
new Country("dj", "Djibouti"),
new Country("do", "Dominica"),
new Country("dr", "Dominican Republic"),
new Country("ec", "Ecuador"),
new Country("eg", "Egypt"),
new Country("es", "El Salvador"),
new Country("ek", "Equatorial Guinea"),
new Country("er", "Eritrea"),
new Country("en", "Estonia"),
new Country("wz", "Eswatini"),
new Country("et", "Ethiopia"),
new Country("fk", "Falkland Islands (Islas Malvinas)"),
new Country("fo", "Faroe Islands"),
new Country("fj", "Fiji"),
new Country("fi", "Finland"),
new Country("fr", "France"),
new Country("fp", "French Polynesia"),
new Country("gb", "Gabon"),
new Country("ga", "Gambia, The"),
new Country("gz", "Gaza Strip"),
new Country("gg", "Georgia"),
new Country("gm", "Germany"),
new Country("gh", "Ghana"),
new Country("gi", "Gibraltar"),
new Country("gr", "Greece"),
new Country("gl", "Greenland"),
new Country("gj", "Grenada"),
new Country("gq", "Guam"),
new Country("gt", "Guatemala"),
new Country("gk", "Guernsey"),
new Country("gv", "Guinea"),
new Country("pu", "Guinea-Bissau"),
new Country("gy", "Guyana"),
new Country("ha", "Haiti"),
new Country("hm", "Heard Island And Mcdonald Islands"),
new Country("vt", "Holy See (Vatican City)"),
new Country("ho", "Honduras"),
new Country("hk", "Hong Kong"),
new Country("hu", "Hungary"),
new Country("ic", "Iceland"),
new Country("in", "India"),
new Country("xo", "Indian Ocean"),
new Country("id", "Indonesia"),
new Country("ir", "Iran"),
new Country("iz", "Iraq"),
new Country("ei", "Ireland"),
new Country("im", "Isle Of Man"),
new Country("is", "Israel"),
new Country("it", "Italy"),
new Country("jm", "Jamaica"),
new Country("jn", "Jan Mayen"),
new Country("ja", "Japan"),
new Country("je", "Jersey"),
new Country("jo", "Jordan"),
new Country("kz", "Kazakhstan"),
new Country("ke", "Kenya"),
new Country("kr", "Kiribati"),
new Country("kn", "Korea, North"),
new Country("ks", "Korea, South"),
new Country("kv", "Kosovo"),
new Country("ku", "Kuwait"),
new Country("kg", "Kyrgyzstan"),
new Country("la", "Laos"),
new Country("lg", "Latvia"),
new Country("le", "Lebanon"),
new Country("lt", "Lesotho"),
new Country("li", "Liberia"),
new Country("ly", "Libya"),
new Country("ls", "Liechtenstein"),
new Country("lh", "Lithuania"),
new Country("lu", "Luxembourg"),
new Country("mc", "Macau"),
new Country("mk", "Macedonia"),
new Country("ma", "Madagascar"),
new Country("mi", "Malawi"),
new Country("my", "Malaysia"),
new Country("mv", "Maldives"),
new Country("ml", "Mali"),
new Country("mt", "Malta"),
new Country("rm", "Marshall Islands"),
new Country("mr", "Mauritania"),
new Country("mp", "Mauritius"),
new Country("mx", "Mexico"),
new Country("fm", "Micronesia, Federated States Of"),
new Country("md", "Moldova"),
new Country("mn", "Monaco"),
new Country("mg", "Mongolia"),
new Country("mj", "Montenegro"),
new Country("mh", "Montserrat"),
new Country("mo", "Morocco"),
new Country("mz", "Mozambique"),
new Country("wa", "Namibia"),
new Country("nr", "Nauru"),
new Country("bq", "Navassa Island"),
new Country("np", "Nepal"),
new Country("nl", "Netherlands"),
new Country("nc", "New Caledonia"),
new Country("nz", "New Zealand"),
new Country("nu", "Nicaragua"),
new Country("ng", "Niger"),
new Country("ni", "Nigeria"),
new Country("ne", "Niue"),
new Country("nf", "Norfolk Island"),
new Country("cq", "Northern Mariana Islands"),
new Country("no", "Norway"),
new Country("mu", "Oman"),
new Country("zn", "Pacific Ocean"),
new Country("pk", "Pakistan"),
new Country("ps", "Palau"),
new Country("um", "Palmyra Atoll"),
new Country("pm", "Panama"),
new Country("pp", "Papua New Guinea"),
new Country("pf", "Paracel Islands"),
new Country("pa", "Paraguay"),
new Country("pe", "Peru"),
new Country("rp", "Philippines"),
new Country("pc", "Pitcairn Islands"),
new Country("pl", "Poland"),
new Country("po", "Portugal"),
new Country("rq", "Puerto Rico"),
new Country("qa", "Qatar"),
new Country("ro", "Romania"),
new Country("rs", "Russia"),
new Country("rw", "Rwanda"),
new Country("tb", "Saint Barthelemy"),
new Country("sh", "Saint Helena, Ascension, And Tristan Da Cunha"),
new Country("sc", "Saint Kitts And Nevis"),
new Country("st", "Saint Lucia"),
new Country("rn", "Saint Martin"),
new Country("sb", "Saint Pierre And Miquelon"),
new Country("vc", "Saint Vincent And The Grenadines"),
new Country("ws", "Samoa"),
new Country("sm", "San Marino"),
new Country("tp", "Sao Tome And Principe"),
new Country("sa", "Saudi Arabia"),
new Country("sg", "Senegal"),
new Country("ri", "Serbia"),
new Country("se", "Seychelles"),
new Country("sl", "Sierra Leone"),
new Country("sn", "Singapore"),
new Country("nn", "Sint Maarten"),
new Country("lo", "Slovakia"),
new Country("si", "Slovenia"),
new Country("bp", "Solomon Islands"),
new Country("so", "Somalia"),
new Country("sf", "South Africa"),
new Country("sx", "South Georgia And South Sandwich Islands"),
new Country("od", "South Sudan"),
new Country("oo", "Southern Ocean"),
new Country("sp", "Spain"),
new Country("pg", "Spratly Islands"),
new Country("ce", "Sri Lanka"),
new Country("su", "Sudan"),
new Country("ns", "Suriname"),
new Country("sv", "Svalbard"),
new Country("sw", "Sweden"),
new Country("sz", "Switzerland"),
new Country("sy", "Syria"),
new Country("tw", "Taiwan"),
new Country("ti", "Tajikistan"),
new Country("tz", "Tanzania"),
new Country("th", "Thailand"),
new Country("tt", "Timor-Leste"),
new Country("to", "Togo"),
new Country("tl", "Tokelau"),
new Country("tn", "Tonga"),
new Country("td", "Trinidad And Tobago"),
new Country("ts", "Tunisia"),
new Country("tu", "Turkey"),
new Country("tx", "Turkmenistan"),
new Country("tk", "Turks And Caicos Islands"),
new Country("tv", "Tuvalu"),
new Country("ug", "Uganda"),
new Country("up", "Ukraine"),
new Country("ae", "United Arab Emirates"),
new Country("uk", "United Kingdom"),
new Country("us", "United States"),
new Country("uy", "Uruguay"),
new Country("uz", "Uzbekistan"),
new Country("nh", "Vanuatu"),
new Country("ve", "Venezuela"),
new Country("vm", "Vietnam"),
new Country("vq", "Virgin Islands"),
new Country("wq", "Wake Island"),
new Country("wf", "Wallis And Futuna"),
new Country("we", "West Bank"),
new Country("wi", "Western Sahara"),
new Country("ym", "Yemen"),
new Country("za", "Zambia"),
new Country("zi", "Zimbabwe"),
new Country("ee", "European Union")
};
}
}
}
Country class
DownloadData Function
The DownloadFunction must retrieve the bulk Factbook JSON and load it into the global variable data where other functions will expect it to be.
// DownloadData : download World Factbook JSON data file
[FunctionName("DownloadData")]
public static bool DownloadData([ActivityTrigger] string name, ILogger log)
{
try
{
// Download factbook.json
String filename = "factbook.json";
if (!LocalTest || !File.Exists(filename))
{
log.LogInformation("Downloading data from World Factbook github");
using (WebClient web = new WebClient())
{
web.DownloadFile(DownloadUrl, filename);
}
}
String json = File.ReadAllText(filename);
data = JObject.Parse(json);
if (!LocalTest) DeleteTempFile(filename);
log.LogInformation("End function DownloadData (success)");
return true;
}
catch (Exception ex)
{
log.LogError("DownloadData: exception - " + ex.Message);
return false;
}
}
DownloadData Function
UploadCountryData
The UploadCountryData function is passed a Country object from the master country list. Using the country object's Key property, the country JSON record can be found in the data object. This fragment of the overall JSON, our country record, goes into a JToken object.We inject three of our properties into the JSON: a source column, which Cosmos DB is expecting in order to set the partition key; a timestamp column, so we can track when this record was captured; and a key property, the country key.
// UploadCountryData : upload World Factbook JSON dataset for one country to blob storage and Cosmos DB
[FunctionName("UploadCountryData")]
public static async Task<bool> UploadCountryData([ActivityTrigger] Country country, ILogger log)
{
log.LogInformation("---- " + country.Name + ": UploadCountryData spawned for country " + country.Name);
JToken token = null;
String json = null;
try
{
token = data["countries"][country.Key]["data"];
// Add properties to the Country JSON.
// source: The Cosmos DB County collection uses /source as the partition key.
// timestamp: timestamp when this data was collected.
// key: country key
token["name"].Parent.AddAfterSelf(new JProperty("source", "Factbook"));
token["name"].Parent.AddAfterSelf(new JProperty("timestamp", DateTime.UtcNow.ToString("U")));
token["name"].Parent.AddAfterSelf(new JProperty("key", country.Key));
json = token.ToString();
}
catch (Exception ex)
{
log.LogInformation("***** Country key : " + country.Key + " not found in JSON data");
log.LogError(ex, "UploadCountryData(" + country.Key + ").blob-upload failed");
return false;
}
PartitionKey parkey = new PartitionKey("Factbook");
if (!UploadCountryData_UploadJson(log, country, json)) return false;
if (!await UploadCountryData_DeletePriorRecords(log, country, parkey)) return false;
if (!await UploadCountryData_AddRecord(log, country, parkey, token)) return false;
if (!UploadCountryData_UploadFlagImage(log, country)) return false;
if (!UploadCountryData_UploadMapImage(log, country)) return false;
return true;
}
UploadCountryData Function
This function has multiple responsibilities, each of which is carried out by a sub-function:
- UploadCountryData_UploadJson: uploads the country JSON to blob storage
- UploadCountryData_DeletePriorRecords: delete any prior database record for the country
- UploadCountryData_AddRecord: adds a database record for the country
- UploadCountryData_UploadFlagImage: retrives country flag image and uploads to blob storage
- UploadCountryData_UploadMapImage: retrieves country map image and uploads to blob storage
UploadCountryData_UploadJson
The UploadCountryData_UploadJson function is responsible for uploading the country JSON to Azure blob storage. For this I am using the PolyCloud.Storage library.// Upload <country>.json to blob storage
private static bool UploadCountryData_UploadJson(ILogger log, Country country, String json)
{
try
{
String filename = country.Key + ".json";
log.LogInformation($"Uploading country data to blob " + filename);
File.WriteAllText(filename, json);
using (Storage storage = Storage.Azure(StorageName, StorageKey))
{
storage.Open();
storage.UploadFile("data", filename);
storage.Close();
}
DeleteTempFile(filename);
return true;
}
catch (Exception ex)
{
log.LogError(ex, "UploadCountryData(" + country.Key + ").blob-upload failed");
return false;
}
}
UploadCountryData_DeletePriorRecords
Prior to adding the country database record, any existing database record for the country is removed. To locate existing records for the country, CreateDocumentQuery (line 11) is called to query for the country name. If any records are found by the query they are deleted with DeleteDocumentAsync (line 32).Because this function is one of many running in parallel, we must account for too many database requests that exceed the throughput rate for the collection. That is why the code has a queryDone flag, is surrounded by a loop, and has sleep logic in its catch handlers. In the event of a Request Rate Too Large Error, the thread will sleep using the RetryAfter property of the DocumentClientException, a computed value optimized for just this pattern of retries. The delete will be attempted up to 3 times in the event of this error.
// If country record already exists in Country collection, remove it
private static async Task<bool> UploadCountryData_DeletePriorRecords(ILogger log, Country country, PartitionKey parkey)
{
// If country record already exists, remove it
log.LogInformation("---- " + country.Key + ": deleting prior Country DB records...");
var query = new SqlQuerySpec("SELECT * FROM Country c WHERE c.name = @name",
new SqlParameterCollection(new SqlParameter[] { new SqlParameter { Name = "@name", Value = country.Name } }));
var existingCountryRecords = client.CreateDocumentQuery<Microsoft.Azure.Documents.Document>(CountryUrl, query, new FeedOptions() { PartitionKey = parkey });
C
bool queryDone = false;
int tries = 0;
if (existingCountryRecords != null)
{
int count = 0;
queryDone = false;
tries = 0;
while (!queryDone)
{
try
{
tries++;
RequestOptions options = new RequestOptions() { PartitionKey = parkey };
if (existingCountryRecords != null)
{
foreach (Microsoft.Azure.Documents.Document c in existingCountryRecords)
{
await client.DeleteDocumentAsync(c.SelfLink, options);
count++;
}
if (count>0) log.LogInformation("---- " + country.Key + ": prior record deleted");
}
queryDone = true;
return true;
}
catch (DocumentClientException de)
{
var statusCode = (int)de.StatusCode;
if ((statusCode == 429 || statusCode == 503) && tries < 3)
{
log.LogInformation(">>>> Error 429/503(de): " + country.Key + " : RETRYING DELETE AFTER SLEEP <<<<");
Thread.Sleep(de.RetryAfter);
}
else
{
log.LogError(de, "UploadCountryData(" + country.Key + ").db-delete failed");
queryDone = true; // exit loop because of hard failure
return false;
}
}
catch (System.AggregateException ae)
{
// See if a request rate too large occurred
if (ae.InnerException.GetType() == typeof(DocumentClientException))
{
var docExcep = ae.InnerException as DocumentClientException;
var statusCode = (int)docExcep.StatusCode;
if ((statusCode == 429 || statusCode == 503) && tries < 3)
{
log.LogInformation(">>>> Error 429/503(ae): " + country.Key + " : RETRYING DELETE AFTER SLEEP <<<<");
Thread.Sleep(docExcep.RetryAfter);
}
else
{
log.LogError(ae, "UploadCountryData(" + country.Key + ").db-delete db failed");
queryDone = true; // exit loop because of hard failure
return false;
}
}
}
} // end while !queryDone
} // end if records-to-delete
return true;
}
UploadCountryData_DeletePriorRecords Function
We can see retries in action when monitoring the function console:
Update Function Performing Retries
UploadCountryData_AddRecord
With any prior country record removed for this country, we are free to add a record with the current data. Like UploadCountryData_DeleteRecord, this follows the same sleep-and-retry-on-error pattern if the request rate is too large.The database record is added with CreateDocumentAsync (line 13), passing the JToken object with the JSON country data.
// Add record to Country collection
private static async Task<bool> UploadCountryData_AddRecord(ILogger log, Country country, PartitionKey parkey, JToken token)
{
bool queryDone = false;
int tries = 0;
while (!queryDone)
{
try
{
log.LogInformation("---- " + country.Key + ": adding database record");
Document doc = await client.CreateDocumentAsync(CountryUrl, token, new RequestOptions() { PartitionKey = parkey });
log.LogInformation("Country record " + country.Key + " created");
queryDone = true;
}
catch (DocumentClientException de)
{
var statusCode = (int)de.StatusCode;
if ((statusCode == 429 || statusCode == 503) && tries < 3)
{
log.LogInformation(">>>> Error 429/503(de): " + country.Key + " : RETRYING ADD AFTER SLEEP <<<<");
Thread.Sleep(de.RetryAfter);
}
else
{
log.LogError(de, "UploadCountryData(" + country.Key + ").db-add failed");
queryDone = true; // exit loop because of hard failure
return false;
}
}
catch (System.AggregateException ae)
{
// See if a request rate too large occurred
if (ae.InnerException.GetType() == typeof(DocumentClientException))
{
var docExcep = ae.InnerException as DocumentClientException;
var statusCode = (int)docExcep.StatusCode;
if ((statusCode == 429 || statusCode == 503) && tries < 3)
{
log.LogInformation(">>>> Error 429/503(ae): " + country.Key + " : RETRYING ADD AFTER SLEEP <<<<");
Thread.Sleep(docExcep.RetryAfter);
}
else
{
log.LogError(ae, "UploadCountryData(" + country.Key + ").db-add db failed");
queryDone = true; // exit loop because of hard failure
return false;
}
}
}
} // end while !queryDone
return true;
UploadCountryData_AddRecord Function
UploadCountryData_UploadFlagImage
The UploadCountryData_UploadFlagImage function has the job of downloading the country flag image from the World Factbook site and uploading it to blob storage. This is skipped if a flag image for the country is already present in blob storage. A constant string in the code has the format of the URL for the flag image, including a placeholder replaced by the 2-letter country code.The image is downloaded with WebClient, and is then uploaded to blob storage using the PolyCloud.Storage library.
// Download country flag image and upload to blob storage as <country>.gif. Only do this if blob does not already exist.
private static bool UploadCountryData_UploadFlagImage(ILogger log, Country country)
{
using (Storage storage = Storage.Azure(StorageName, StorageKey))
{
String flagUrl = String.Format(FlagUrlFormat, country.Code.ToUpper());
String flagFilename = country.Key + ".gif";
if (!storage.FileExists(ContainerName, flagFilename))
{
try
{
// Download flag image file
log.LogInformation($"Downloading country flag from " + flagUrl);
using (WebClient web = new WebClient())
{
web.DownloadFile(flagUrl, flagFilename);
}
// Upload to blob storage
log.LogInformation($"Uploading country flag to blob " + flagFilename);
storage.Open();
storage.UploadFile("data", flagFilename);
storage.Close();
DeleteTempFile(flagFilename);
}
catch (Exception ex)
{
// Flag image not available
log.LogError(ex, "UploadCountryData(" + country.Key + ").download-flag failed");
return false;
}
}
return true;
}
}
UploadCountryData_UploadFlagImage Function
Here's a link to a country flag image in blob storage.
UploadCountryData_UploadMapImage
The UploadCountryData_UploadMapImage function is just like UploadCountryData_UploadFlagImage, except it is retrieving and uploading a map image. All the mechanics are otherwise the same.// Download country map image and upload to blob storage as <country>.gif. Only do this if blob does not already exist.
private static bool UploadCountryData_UploadMapImage(ILogger log, Country country)
{
using (Storage storage = Storage.Azure(StorageName, StorageKey))
{
String mapUrl = String.Format(MapUrlFormat, country.Code.ToUpper());
String mapFilename = country.Key + "-map.gif";
if (!storage.FileExists(ContainerName, mapFilename))
{
try
{
// Download flag image file
log.LogInformation($"Downloading country map from " + mapUrl);
using (WebClient web = new WebClient())
{
web.DownloadFile(mapUrl, mapFilename);
}
// Upload to storage
log.LogInformation($"Uploading country map to blob " + mapFilename);
storage.Open();
storage.UploadFile("data", mapFilename);
storage.Close();
DeleteTempFile(mapFilename);
}
catch (Exception ex)
{
// Map image not available
log.LogError(ex, "UploadCountryData(" + country.Key + ").download-map failed");
return false;
}
}
return true;
}
}
UploadCountryData_UploadMapImage Function
Here's a link to a country map image in blob storage.
https://worldfactbook.blob.core.windows.net/data/peru-map.gif
Querying the Database
With the data retrieved and loaded, we can verify it by querying the Cosmos DB database using the Azure Portal. If we visit the Data Explorer in the Azure Portal, we can go to full screen mode and click New SQL Query. Now we can query the data any way we wish. The first thing we want to see is that we loaded the data we expected to. Here's a query to list each country, showing the key and name:SELECT c.key, c.name from c order by c.key
Data Explorer - Enumerating Countries
We have countries from Afghanistan to Zimbabwe in the results, a good sign. Now to look at the entire record for a country to make sure the complete JSON country record is intact.
SELECT * FROM c WHERE c.name='Greece'
SELECT * FROM c WHERE c.name='Greece'
Data Explorer - Country Record
The Country records look like we expect them to, with lots of detail. We also see the three properties we injected into the JSON: key, timestamp, and source.
Query by Location
And now, finally, we can do things with the data. For example, let's say we wanted to find all countries located in the Caribbean. We can do that with this query:select c.name, c.geography.location from c where contains (c.geography.location, 'Caribbean') order by c.name
The results match our expectations: the results span from Anguilla, Aruba, Barbados... all the way to ...Venezuela, Virgin Islands.
Location Query Results
Query to Rank by Population
Similarly, we can query for population. Let's say we wanted to rank countries by population, from largest to smallest. We can do that with this query:SELECT c.name, c.people.population.total FROM c ORDER BY c.people.population.total desc
Population Query Results
Query by Language
Finally, let's query for counties whose primary spoken langage is French. We can do that with this query:
SELECT c.name, c.people.languages FROM c
WHERE c.people.languages.language[0].name='French' ORDER BY c.name
Language Query Results
In Conclusion
Today in Part 1 we built the back-end of a world country data repository. Source code on github
We've built the code to pull down public-domain CIA World Factbook data and create our own country repository on Azure; including a detailed country record, flag image, and map image for 260 countries. To hold the data we create both a blob storage container and a Cosmos DB country collection.
We used Azure Durable Functions to retrieve and store the latest data, and used a Timer Trigger to schedule updates each Saturday. In our code, we used the fan-out/fan-in pattern to extract country data in parallel.
In Part 2, we will create the front end, including an API and a web site. We'll also be using Azure Functions for that.
1 comment:
Informative post. Thanks for sharing
Mobile Solution For Azure
Post a Comment