Make sure to sort dataframes

Since database queries aren't necessarily in the same order every time
the results need to be sorted to compare them

Author: SimonKrughoff

02_Intermediate_TAP_Query.ipynb

@@ -87,6 +87,26 @@
     "warnings.filterwarnings('ignore')"
    ]
   },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "In general the order of results from database queries cannot be assumed to be the same every time.\n",
+    "This function sorts the data so we can compare the result dataframes even if the records are not in the same order from the query."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "def sort_dataframe(df, sort_key='objectid'):\n",
+    "    df = df.sort_values('objectId')\n",
+    "    df.set_index(np.array(range(len(df))), inplace=True)  # Since we are sorting, we need to reset the incremental index as well\n",
+    "    return df"
+   ]
+  },
   {
    "cell_type": "markdown",
    "metadata": {},
@@ -303,8 +323,8 @@
    "source": [
     "# Convert the results to pandas data frames and  assert that the\n",
     "# contents of the two tables are identical\n",
-    "assert_frame_equal(results.to_table().to_pandas(),\n",
-    "                   results1.to_table().to_pandas())"
+    "assert_frame_equal(sort_dataframe(results.to_table().to_pandas()),\n",
+    "                   sort_dataframe(results1.to_table().to_pandas()))"
    ]
   },
   {
@@ -827,7 +847,7 @@
     "# Assert that the results are the same as obtained from\n",
     "# executing synchronous queries\n",
     "assert len(async_results) == 14424 \n",
-    "assert_frame_equal(results, async_results.to_table().to_pandas())"
+    "assert_frame_equal(sort_dataframe(results), sort_dataframe(async_results.to_table().to_pandas()))"
    ]
   },
   {
@@ -848,8 +868,8 @@
     "retrieved_job = retrieve_query(job.url)\n",
     "retrieved_results = retrieved_job.fetch_result()\n",
     "assert len(retrieved_results) == 14424\n",
-    "assert_frame_equal(retrieved_results.to_table().to_pandas(),\n",
-    "                   async_results.to_table().to_pandas())"
+    "assert_frame_equal(sort_dataframe(retrieved_results.to_table().to_pandas()),\n",
+    "                   sort_dataframe(async_results.to_table().to_pandas()))"
    ]
   },
   {