The official blog of the Engineering team at LinkedIn
Real-time analytics on network flow data with Apache PinotThe LinkedIn infrastructure has thousands of services serving millions of queries per second. At this scale, having tools that provide observability into the LinkedIn infrastructure is imperative to ensure that issues in our infrastructure are quickly detected, diagnosed, and remediated. This level of visibility helps prevent the occurrence of outages so we can deliver the best experience for our members. To provide observability, there are various data points that need to be collected, such as metrics, events, logs, and flows. Once collected, the data points can then be processed […]
09/13/22
Feathr joins LF AI & Data FoundationCo-authors: Hangfei Lin, Jinghui Mo We’re excited to announce today that Feathr is joining LF AI & Data, the Linux Foundation’s umbrella foundation supporting open source innovation in artificial intelligence (AI) and data. Feathr is a feature store that simplifies machine learning (ML) feature serving and improves developer productivity. “We're excited to welcome Feathr to LF AI & Data and for it to be part of our technical project portfolio (41 projects and growing) with a community of over 17K developers,” said Dr. Ibrahim Haddad, Executive Director of LF AI & Data. “We aim to […]
09/12/22
Career stories: Rejoining LinkedIn to scale our media infrastructureOriginally from Argentina, systems & infrastructure engineering leader Federico was a founding member of the Media Infrastructure team in 2015. Now based in Bellevue, Wash., Federico shares how his supportive mentor, LinkedIn’s “sweet spot” scale, and the distinctive engineering challenges here ultimately brought him back to LinkedIn in 2019. My love for engineering started in my home country of Argentina. After working as an engineer in a corporate setting for a few years, I decided to start my own company focused on custom software development. I loved the interesting problems I […]
09/06/22
Operating system upgrades at LinkedIn’s scaleCo-authors: Hengyang Hu, Dinesh Dhakal, Kalyanasundaram Somasundaram Introduction Completing recurring operating system (OS) upgrades on time and without impacting users can be challenging. For LinkedIn, completing these upgrades at a massive scale has its own complexities as we’re often facing multiple upgrades. To secure our platform and protect our members’ data, we needed a fast and reliable OS upgrade framework with little to no human intervention. In this blog, we’ll introduce a newly developed system, Operating System Upgrade Automation (OSUA), which allows LinkedIn to scale […]
08/31/22
Challenges and practical lessons from building a deep-learning-based ads CTR prediction modelCo-authors: Ruoyan Wang, Sirou Zhu, Chengming Jiang Introduction At LinkedIn, our ads business is powered by click-through-rate (CTR) prediction, a core machine learning model. CTR prediction estimates the probability of clicks between a LinkedIn member and a potential advertisement. That probability is then used for ads auctions, which decide the order of ads being displayed to members. A better CTR model can enhance the member and advertiser experience by bringing more relevant ads and more efficient advertiser budget spending. In the past, we predicted ads CTR through a GLMix […]
08/29/22
Improving Post Search at LinkedInCo-authors: Amol Chandla, Jagadeesan Sundaresan, Siddharth Pratap Singh, Saipriya Kumar, and Anand Kishore At LinkedIn Search, we strive to serve results that are most relevant to a member’s query, from a wide variety of verticals such as jobs they may be interested in, people they may want to connect with, or posts that are trending within their industry. Post search saw strong organic growth in 2020, with a 35% year-over-year increase in user engagement. As we watched content continue to grow and diversify on the platform, the Flagship Search team saw an opportunity to improve the […]
08/25/22
REACH turns five - Celebrating the power of apprenticeshipsEvery tech company wants to hire great talent, but many have historically hired based on pedigree -- where candidates went to school, where they worked and who they know. However, there's a pool of qualified talent who have followed non-traditional paths but may have been overlooked in traditional recruiting pipelines. Five years ago, we piloted our REACH program to help solve this problem by providing those with non-traditional software engineering backgrounds the opportunity to get their foot in the door and develop their coding skills. What began as a small six-month program […]
08/24/22
Improving Post Search at LinkedInCo-authors: Amol Chandla, Jagadeesan Sundaresan, Siddharth Pratap Singh, Saipriya Kumar, and Anand Kishore At LinkedIn Search, we strive to serve results that are most relevant to a member’s query, from a wide variety of verticals such as jobs they may be interested in, people they may want to connect with, or posts that are trending within their industry. Post search saw strong organic growth in 2020, with a 35% year-over-year increase in user engagement. As we watched content continue to grow and diversify on the platform, the Flagship Search team saw an opportunity to improve the […]
08/24/22
Applying multitask learning to AI models at LinkedInCo-authors: Ji Yan, Sen Zhou, Dansong Zhang, Anastasiya Karpovich Introduction Multitask learning, a subfield of machine learning, aims to accomplish multiple learning tasks at the same time by exploiting commonalities and differences across tasks. Traditionally, engineers have built separate AI models to accomplish each task at hand. While this allows a simple boundary of ownership, the isolated models lack the ability to share any learnings gathered from training the models. For example, new machine learning (ML) tasks often suffer from the cold start problem due to a lack of […]
08/19/22
Career Stories: Breaking barriers with LinkedInAfter interning with us, Beatrix resonated with the culture and community she found at LinkedIn, and rejoined us post-undergrad. As she continues exploring her passion for frontend (UI) and accessibility engineering, she shares why launching her career with LinkedIn is one of the best decisions she has made. From intern to engineer In 2019, my career with LinkedIn started with a UI (frontend) engineering internship in the San Francisco office. I had done a bit of user experience (UX) work before, so I was excited that there was a specific frontend role for interns at LinkedIn. I […]
08/09/22
Measuring downstream impact on social networks by using an attribution frameworkCo-authors: Qiannan Yin, Derek Koh, and Jenny Wu The network effect in social networks increases the complexity of conducting analysis, as the actions of members have the ability to impact others within the network. For instance, a member posting an article on LinkedIn could result in actions downstream where other members see the post and interact with it by means of liking, commenting, or sharing. These downstream actions make it harder to measure the overall impact when we try to combine the primary impact from the posters and the downstream impact from those downstream actions, […]
07/21/22
Measuring marketing incremental impacts beyond last click attributionCo-authors: Maggie Zhang, Joyce Chen, and Ming Wu What’s my ROI? In every company, there’s a fundamental need to understand the impacts of marketing campaigns. You want to be able to measure how many incremental conversions different channels and touchpoints are successfully driving. The best practice of A/B tests at individual level is not applicable in traditional channels such as TV ads, radios, or billboards. Even in digital marketing channels, new regulations and public awareness for data privacy have made A/B testing on third party platforms, which require transferring user […]
07/14/22
Career stories: Next plays, jungle gyms, and PythonSince she was a child, Deepti has been motivated to help people. This drive led her on a career journey with many pivots and moves — akin to navigating a children’s jungle gym — between industries and around the world. Based in Bangalore, this biomedical engineer turned data scientist shares how LinkedIn helped her gain new technical skills, dive into meaningful work, and grow. Growing up in Mumbai, India, I always imagined myself in a career where I could give back. I once dreamed of becoming a neurosurgeon, but early in my career, I took a different path and earned a bachelor’s in […]
06/29/22
Welcome to the Inaugural LinkedIn-Cornell grant recipientsAt LinkedIn, we're always looking for ways to improve our platform for members and customers and believe collaborating with academia can advance these efforts. That is why we are excited to announce the first group of grant recipients from our partnership with the Cornell Ann S. Bowers College of Computing and Information Science (Cornell Bowers CIS). These grants provide financial resources to Ph.D. students and faculty for research and amplify efforts in diversity, equity, and inclusion within Cornell Bowers CIS. The inaugural eight research grants cover areas from training the […]
06/28/22
Migration madness: How to navigate the chaos of large cross-team initiatives towards a common goalCo-authors: Lily Wittle and Cathy Ji Introduction The Gateway-as-a-Platform (GaaP) team at LinkedIn builds infrastructure to support our secure third-party integrations for our customers. LinkedIn features that require sending or receiving data from external sources, such as importing email contacts and sharing LinkedIn posts as tweets, use our infrastructure. In 2020, we sought to deprecate our legacy deployment backend, which impacted every third-party integration at the company, and spanned use cases across dozens of engineering teams. Migrating 600+ use cases to our new […]
06/27/22
INWED2022: How Skills Can Help Shape Your Future in Engineering Today is an exciting day for female engineers! June 23 marks the ninth anniversary of International Women in Engineering Day (INWED). Developed by the Women’s Engineering Society (WES), this annual day of recognition plays a vital role in encouraging more young women to explore the benefits of an engineering career and showing the progress made in the field. Women continue to be underrepresented in engineering, with only 23% of those who list software engineer as their title on LinkedIn being a woman. This gender imbalance can lead to missed opportunities for innovation within the […]
06/23/22
LinkedIn’s journey to Java 11Introduction At LinkedIn, we are committed to deliver a best-in-class platform experience for our members. One of the technologies that we use to do that is Java, an object-oriented programming language that produces software for multiple platforms. We are a huge consumer of Java, running over 1000 Java applications on 320,000+ hosts. Based on the promised performance improvements of the G1 garbage collector (G1GC) and other well-documented benefits captured in this article, we made the decision to begin our journey from Java 8 to Java 11. In this blog post, we will discuss LinkedIn […]
06/16/22
Under the hood: How we built API versioning for LinkedIn Marketing APIsCo-authors: Nihit Purwar, Demian Lessa, and Shruthi Jeganathan At LinkedIn, the goal of our API team is to create a world-class platform that enables partners to build, run, and grow sustainable businesses. Developers use our Marketing APIs to build new and innovative solutions that complement LinkedIn Marketing Solutions and enable marketers to achieve more. While the program has seen success, it has also experienced some growing pains as our approach to change management did not always take into account all of the potential impacts to our partners and customers. As our business […]
06/14/22
Towards data quality management at LinkedInCo-authors: Liangzhao Zeng, Ting Yu (Cliff) Leung, Jimmy Hong, and Kevin Lau Introduction Data is at the heart of all our products and decisions at LinkedIn and the quality of our data is vital to our success. While not an uncommon problem, our scale, hundreds of thousands of pipelines and streams as well as over an exabyte of data in our data lake alone, presents unique challenges. Most pipelines have complex logic involving a multitude of upstream datasets and common data quality issues in these datasets can be broadly classified into two categories: metadata and data semantic. […]
06/09/22
Overcoming challenges with Linux cgroups memory accountingIntroduction LinkedIn's de facto search solution, Galene, is a Search-as-a-Service infrastructure that powers a multitude of search products at LinkedIn, from member-facing searches (such as searching for jobs or other members) to internal index searches. Galene’s responsiveness and reliability are paramount as it caters to many critical features. This post discusses debugging an issue where the hosts ran out of memory and became inaccessible, even though the applications are limited by cgroups. We’ll cover memory accounting in cgroups and how it is not always straightforward when […]