By Ricardo Baeza-Yates, PhD
July 2, 2020
For decades, mathematical optimization has been widely used by companies of all sizes and stripes to address their complex business problems. The secret to mathematical optimization’s staying power is that it has consistently demonstrated that it is capable of generating optimal solutions to large-scale, real-world business problems – and has thereby produced significant business value.
Surprisingly, though, many data scientists lack an understanding of what mathematical optimization is and what it can do. Why is this the case?
The simple answer is: They probably haven’t been exposed to it. Many data scientists study either computer science or statistics at the university level (while a good number of them are actually self-taught), and their advanced analysis tool of choice is machine learning. Unless they have actually taken some courses in mathematical programming (which most of them, I suspect, have not), they are probably not familiar with mathematical optimization tools and techniques.
In the past, mathematical optimization and machine learning developed in and existed in silos – and these technological tools (as well as their practitioners) did not work together.
But these silos are being broken down and the paradigm is changing: A growing number of companies are starting to use mathematical optimization in combination with machine learning – and seeing how these two technologies can complement and enhance each other.
Fundamentally, this relationship can work in three ways:
This third approach is especially intriguing for data scientists as it allows them to truly leverage mathematical optimization’s problem-solving power to improve their current machine-learning applications.
But – one might wonder – why would data scientists want to use mathematical optimization to address their problems? What can mathematical optimization offer that machine learning can’t?
In this article, I will discuss the three key benefits that data scientists can realize by using mathematical optimization in their applications.
One of the shortcomings of machine learning solutions is that, although they can take raw data and produce sophisticated output, they usually don’t consider the semantics of the problem. This means that, in many instances, machine learning applications have to learn things that we already know. For example, natural language processing: A machine learning application has to process and understand so many basic linguistic structures and meanings (which we already know) – and it takes some time for the machine learning application just to grasp everything.
With mathematical optimization, you can add semantics to problems in the form of constraints (essentially rules and restrictions) coded into a formal model. These semantics provide a structure that helps ensure that your subject matter expertise is embedded in the model and that the solutions you generate are not unbound.
It is true that with machine learning, you can also add semantics into the application, but this is not a simple process and needs to be done on an ad hoc basis – as these semantics are not part of a given method. Mathematical optimization gives you the capability to automatically insert semantics (and thus your subject matter expertise) to your applications – and this is tremendously valuable for data scientists.
Most data scientists use machine learning as a black box: They have little idea of what’s happening inside their applications and are likely unable to explain why these applications are generating specific predictions. Data scientists can conduct features analysis and find out which features are the most important factors – but they do not possess a true causal understanding of why the machine learning-based prediction is what it is.
In contrast, the output of mathematical optimization applications is interpretable. Mathematical optimization applications are built on models, which have feature weights that are associated with particular variables and constraints. This makes it possible to analyze the model and pinpoint which variables and constraints are causing it to generate a specific solution.
In this way, mathematical optimization can be used to improve the interpretability of predictions. Having interpretable solutions that can be understood and trusted – by data scientists and subject matter experts alike – is vital, especially when it comes to “life-or-death” classification problems like predicting health problems such as heart disease or diabetes.
By using mathematical optimization to address classification and prediction problems, data scientists can gain a causal understanding of what is happening in the system and gain more confidence in its output.
There is a degree of uncertainty that is baked into any machine-learning-based prediction. It is, after all, a best guess about future conditions based on historical data. Data scientists try to minimize the errors associated with these predictions – to make sure that they can make the best possible business decisions based on these predictions.
The problem is that data scientists often do not know how far away these predictions are from the minimum total error because traditional predictive models cannot properly handle constraints and typically rely on heuristics. By using these heuristics, data scientists don’t know how close they are to the minimum total error.
By taking the original problem and feeding it into a mathematical optimization solver, data scientists can determine exactly how close they are to an optimal solution. For example, if the total error derived from a heuristics approach might be 60% away from the minimum total error, that means the heuristic is not an effective predictive approach.
By using mathematical optimization to determine this optimality gap, data scientists can bound and minimize the error of their predictions. This enables them to ensure that their predictions are accurate.
Many data scientists today use machine learning like a cannon that they fire at every problem, but as Confucius said, “You shouldn’t use a cannon to kill a fly.” Indeed, machine learning is a powerful technological tool, but it is not necessarily the best solution for every business problem.
To address the many different complex problems in our world today and enhance their machine learning applications, I would encourage data scientists to add new advanced analytics technologies into their toolboxes.
Although mathematical optimization is an established and mature technology with a history stretching back more than 70 years, it is new tool for many data scientists – and I believe that we are only just starting to discover all the possible ways to integrate and utilize mathematical optimization and machine learning together.
Choose the evaluation license that fits you best, and start working with our Expert Team for technical guidance and support.
Request free trial hours, so you can see how quickly and easily a model can be solved on the cloud.
|_biz_flagsA||1 year||A Cloudflare cookie set to record users’ settings as well as for authentication and analytics.|
|_biz_pendingA||1 year||A Cloudflare cookie set to record users’ settings as well as for authentication and analytics.|
|_biz_sid||30 minutes||This cookie is set by Bizible, to store the user's session id.|
|ARRAffinity||session||ARRAffinity cookie is set by Azure app service, and allows the service to choose the right instance established by a user to deliver subsequent requests made by that user.|
|ARRAffinitySameSite||session||This cookie is set by Windows Azure cloud, and is used for load balancing to make sure the visitor page requests are routed to the same server in any browsing session.|
|cookielawinfo-checkbox-advertisement||1 year||Set by the GDPR Cookie Consent plugin, this cookie is used to record the user consent for the cookies in the "Advertisement" category .|
|cookielawinfo-checkbox-analytics||11 months||This cookie is set by GDPR Cookie Consent plugin. The cookie is used to store the user consent for the cookies in the category "Analytics".|
|cookielawinfo-checkbox-functional||11 months||The cookie is set by GDPR cookie consent to record the user consent for the cookies in the category "Functional".|
|cookielawinfo-checkbox-necessary||11 months||This cookie is set by GDPR Cookie Consent plugin. The cookies is used to store the user consent for the cookies in the category "Necessary".|
|cookielawinfo-checkbox-others||11 months||This cookie is set by GDPR Cookie Consent plugin. The cookie is used to store the user consent for the cookies in the category "Other.|
|cookielawinfo-checkbox-performance||11 months||This cookie is set by GDPR Cookie Consent plugin. The cookie is used to store the user consent for the cookies in the category "Performance".|
|CookieLawInfoConsent||1 year||Records the default button state of the corresponding category & the status of CCPA. It works only in coordination with the primary cookie.|
|elementor||never||This cookie is used by the website's WordPress theme. It allows the website owner to implement or change the website's content in real-time.|
|JSESSIONID||session||New Relic uses this cookie to store a session identifier so that New Relic can monitor session counts for an application.|
|__cf_bm||30 minutes||This cookie, set by Cloudflare, is used to support Cloudflare Bot Management.|
|_biz_nA||1 year||Bizible sets this cookie to remember users’ settings as well as for authentication and analytics.|
|_biz_uid||1 year||This cookie is set by Bizible, to store user id on the current domain.|
|_hjAbsoluteSessionInProgress||30 minutes||Hotjar sets this cookie to detect a user's first pageview session, which is a True/False flag set by the cookie.|
|_mkto_trk||2 years||This cookie is set by Marketo. This allows a website to track visitor behavior on the sites on which the cookie is installed and to link a visitor to the recipient of an email marketing campaign, to measure campaign effectiveness. Tracking is performed anonymously until a user self-identifies by submitting a form.|
|bcookie||1 year||LinkedIn sets this cookie from LinkedIn share buttons and ad tags to recognize browser ID.|
|bscookie||1 year||LinkedIn sets this cookie to store performed actions on the website.|
|doc_langsBB||1 year||Documentation system cookie|
|doc_version||1 year||Documentation system cookie|
|lang||session||LinkedIn sets this cookie to remember a user's language setting.|
|lidc||1 day||LinkedIn sets the lidc cookie to facilitate data center selection.|
|UserMatchHistory||1 month||LinkedIn sets this cookie for LinkedIn Ads ID syncing.|
|whova_client_id||10 years||Event agenda system cookie|
|_gat_UA-5909815-1||1 minute||A variation of the _gat cookie set by Google Analytics and Google Tag Manager to allow website owners to track visitor behaviour and measure site performance. The pattern element in the name contains the unique identity number of the account or website it relates to.|
|_an_uid||7 days||6Sense Cookie|
|_BUID||1 year||This cookie, set by Bizible, is a universal user id to identify the same user across multiple clients’ domains.|
|_ga||2 years||The _ga cookie, installed by Google Analytics, calculates visitor, session and campaign data and also keeps track of site usage for the site's analytics report. The cookie stores information anonymously and assigns a randomly generated number to recognize unique visitors.|
|_ga_*||1 year 1 month 4 days||Google Analytics sets this cookie to store and count page views.|
|_gat_UA-*||1 minute||Google Analytics sets this cookie for user behaviour tracking.|
|_gcl_au||3 months||Provided by Google Tag Manager to experiment advertisement efficiency of websites using their services.|
|_gd_session||4 hours||This cookie is used for collecting information on users visit to the website. It collects data such as total number of visits, average time spent on the website and the pages loaded.|
|_gd_visitor||2 years||This cookie is used for collecting information on the users visit such as number of visits, average time spent on the website and the pages loaded for displaying targeted ads.|
|_gid||1 day||Installed by Google Analytics, _gid cookie stores information on how visitors use a website, while also creating an analytics report of the website's performance. Some of the data that are collected include the number of visitors, their source, and the pages they visit anonymously.|
|_hjFirstSeen||30 minutes||Hotjar sets this cookie to identify a new user’s first session. It stores the true/false value, indicating whether it was the first time Hotjar saw this user.|
|_hjIncludedInSessionSample_*||2 minutes||Hotjar cookie that is set to determine if a user is included in the data sampling defined by a site's daily session limit.|
|_hjRecordingEnabled||never||Hotjar sets this cookie when a Recording starts and is read when the recording module is initialized, to see if the user is already in a recording in a particular session.|
|_hjRecordingLastActivity||never||Hotjar sets this cookie when a user recording starts and when data is sent through the WebSocket.|
|_hjSession_*||30 minutes||Hotjar cookie that is set when a user first lands on a page with the Hotjar script. It is used to persist the Hotjar User ID, unique to that site on the browser. This ensures that behavior in subsequent visits to the same site will be attributed to the same user ID.|
|_hjSessionUser_*||1 year||Hotjar cookie that is set when a user first lands on a page with the Hotjar script. It is used to persist the Hotjar User ID, unique to that site on the browser. This ensures that behavior in subsequent visits to the same site will be attributed to the same user ID.|
|_hjTLDTest||session||To determine the most generic cookie path that has to be used instead of the page hostname, Hotjar sets the _hjTLDTest cookie to store different URL substring alternatives until it fails.|
|6suuid||2 years||6Sense Cookie|
|AnalyticsSyncHistory||1 month||LinkedIn cookie|
|BE_CLA3||1 year 1 month 4 days||BrightEdge sets this cookie to enable data aggregation, analysis and report creation to assess marketing effectiveness and provide solutions for SEO, SEM and website performance.|
|CONSENT||2 years||YouTube sets this cookie via embedded youtube-videos and registers anonymous statistical data.|
|dj||10 years||DemandJump cookie|
|djaimid.a28e||2 years||DemandJump cookiean|
|djaimses.a28e||30 minutes||DemandJump cookie|
|li_gc||5 months 27 days||LinkedIn Cookie|
|ln_or||1 day||LinkedIn Cookie|
|vuid||2 years||Vimeo installs this cookie to collect tracking information by setting a unique ID to embed videos to the website.|
|__adroll||1 year 1 month||This cookie is set by AdRoll to identify users across visits and devices. It is used by real-time bidding for advertisers to display relevant advertisements.|
|__adroll_fpc||1 year||AdRoll sets this cookie to target users with advertisements based on their browsing behaviour.|
|__adroll_shared||1 year 1 month||Adroll sets this cookie to collect information on users across different websites for relevant advertising.|
|__ar_v4||1 year||This cookie is set under the domain DoubleClick, to place ads that point to the website in Google search results and to track conversion rates for these ads.|
|_fbp||3 months||This cookie is set by Facebook to display advertisements when either on Facebook or on a digital platform powered by Facebook advertising, after visiting the website.|
|fr||3 months||Facebook sets this cookie to show relevant advertisements to users by tracking user behaviour across the web, on sites that have Facebook pixel or Facebook social plugin.|
|IDE||1 year 24 days||Google DoubleClick IDE cookies are used to store information about how the user uses the website to present them with relevant ads and according to the user profile.|
|li_sugr||3 months||LinkedIn sets this cookie to collect user behaviour data to optimise the website and make advertisements on the website more relevant.|
|test_cookie||15 minutes||The test_cookie is set by doubleclick.net and is used to determine if the user's browser supports cookies.|
|VISITOR_INFO1_LIVE||5 months 27 days||A cookie set by YouTube to measure bandwidth that determines whether the user gets the new or old player interface.|
|YSC||session||YSC cookie is set by Youtube and is used to track the views of embedded videos on Youtube pages.|
|yt-remote-connected-devices||never||YouTube sets this cookie to store the video preferences of the user using embedded YouTube video.|
|yt-remote-device-id||never||YouTube sets this cookie to store the video preferences of the user using embedded YouTube video.|
|yt.innertube::nextId||never||This cookie, set by YouTube, registers a unique ID to store data on what videos from YouTube the user has seen.|
|yt.innertube::requests||never||This cookie, set by YouTube, registers a unique ID to store data on what videos from YouTube the user has seen.|