Puneet Ghanshani

Chief Architect at Microsoft

16 minute read

As consulting architects, we often help customers facing challenges in managing large volumes of data across different systems and departments. This can lead to inefficiencies and difficulty accessing accurate insights. For example, in asset-heavy industries, identifying correct maintenance schedules or team responsibilities can be a struggle, causing delays and errors. A knowledge graph can link and organize this data, enabling easier access to insights and streamlining decision-making.

Knowledge graphs improve search accuracy by enabling semantic and relationship-aware retrieval, going beyond keyword matching to interpret queries conceptually. In industries like manufacturing, energy, utilities, and transportation, this boosts operational efficiency by accelerating decision-making and problem resolution.

Named Entity Recognition (NER), an NLP technique, can automatically build knowledge graphs by identifying key entities (e.g., equipment, manufacturer, location). Entity disambiguation ensures accurate mapping of similar names (e.g., “Model X-200”), preventing errors in maintenance or procurement.

Constructing a Knowledge Graph Using NER

1. Extracting Entities from Text

NER identifies and classifies key entities in a given text. For example, in an industrial setting, consider the sentence:

The GE Turbine Model X-200, installed at the Texas wind farm in 2019, requires maintenance due to abnormal vibration levels.

A well-trained NER model extracts:

  • Equipment → “GE Turbine Model X-200”
  • Manufacturer → “GE”
  • Location → “Texas wind farm”
  • Event/Issue → “abnormal vibration”
  • Date → “2019”
  • Process → “maintenance”

2. Identifying Relationships Between Entities

Once entities are extracted, relationships between them must be identified using:

  • Syntactic and dependency parsing (analyzing sentence structure)
  • Knowledge base matching (comparing with existing maintenance databases)
  • Rule-based or AI-driven relationship extraction (training models to infer relationships)

From the example:

  • (GE Turbine Model X-200) → (Manufactured by) → (GE)
  • (GE Turbine Model X-200) → (Installed at) → (Texas wind farm)
  • (GE Turbine Model X-200) → (Has issue) → (Abnormal vibration)
  • (Abnormal vibration) → (Requires) → (Maintenance)

3. Constructing the Graph

After extracting entities and relationships, they can be represented as a graph:

    (GE Turbine X-200)
          |
    [Manufactured by]
          |
       (GE)
          |
    [Installed at]
          |
  (Texas Wind Farm) ---- [Has issue] ----> (Abnormal Vibration)
          |
    [Requires]
          |
    (Maintenance)

Each entity becomes a node, and each relationship forms an edge, creating a structured knowledge representation.

Enhancing Search Accuracy and Contextual Understanding

Knowledge graphs excel at semantic search by capturing the contextual meaning of asset data:

  1. Disambiguation Through Entity Resolution
    • Industrial terms often have multiple meanings. For instance, “Jordan” could refer to a location, a person, or a brand name. Advanced techniques (e.g., graph embeddings like DeepWalk or EDEGE) analyze the surrounding context and relationships to correctly interpret ambiguous terms.
  2. Relationship-Aware Retrieval
    • Queries such as “Which turbines have shown abnormal vibration in the past three months?” leverage the relationship density between entities. Graph databases with index-free adjacency enable quick traversal of connected nodes (turbines, vibration logs, dates), returning highly relevant results without sifting through thousands of unconnected records.
  3. Cross-Domain Linking
    • Asset data often resides in siloed systems—maintenance logs, engineering files, and procurement records. By linking these sources through entity attributes (e.g., serial number, installation date), the knowledge graph provides a unified view for complex queries like “Find all turbines installed before 2020 that have had multiple vibration-related maintenance incidents.”
  4. Hybrid Retrieval-Augmented Generation (RAG)
    • In advanced AI systems, a dual approach combines vector-based retrieval (for textual similarity) with graph-based retrieval (for structured context). For example, a question like “Are there known correlations between turbine rotor size and frequency of vibration issues?” can retrieve relevant text passages (from maintenance manuals) as well as subgraphs connecting rotor size, turbine model, and past incidents, producing a more factually grounded response.

Challenges in Maintaining Large-Scale Knowledge Graphs

Even with strong benefits in semantic search and disambiguation, scaling and maintaining a knowledge graph in asset-intensive industries involves managing data quality, continuous updates, and robust security.

Ensuring Data Quality and Consistency
Asset data can originate from sensor readings, operator logs, or third-party vendors. Conflicting or incomplete entries risk generating inaccurate insights. Automated validation checks for anomalies (e.g., a vibration reading contradicting a maintenance record), while data provenance ensures every piece of information has a traceable source. Where discrepancies appear, human reviewers step in to finalize decisions.

Scaling Knowledge Graphs for Industrial Assets
A large wind farm or manufacturing plant may contain hundreds of thousands of assets. Relationship queries grow exponentially as the graph expands. Distributed graph databases and partitioning (by location or asset type) keep response times low, while index-free adjacency accelerates multi-hop traversals crucial for complex analytics.

Continuous Updates
Industrial environments are dynamic—equipment may be upgraded, replaced, or relocated. Event-driven architectures (e.g., Apache Kafka) feed these changes into the knowledge graph in near real-time, ensuring that search results and analytics remain accurate and up-to-date.

Security and Usability
Protecting proprietary asset data is critical. Implementing role-based access controls and encryption defends against unauthorized access, whereas interactive dashboards and simplified query interfaces make the system more accessible to maintenance engineers, analysts, and decision-makers.

Real-World Examples

1. Wind Turbine Maintenance

Combining NER with semantic search functionalities offers tangible benefits in asset management. Revisiting the GE Turbine Model X-200 scenario:

  1. A maintenance engineer queries:
    “List all turbines showing abnormal vibration over the last month and any correlated weather factors.”
  2. The knowledge graph quickly retrieves:
    • Turbines of the same model installed within the same farm.
    • Historical logs documenting similar vibration anomalies.
    • Local weather data (high winds, temperature extremes) linked to each turbine’s operational timeline.
  3. Using relationship-aware retrieval, the system highlights two additional turbines likely to exhibit similar issues. This proactive maintenance capability helps reduce costly downtime and prevents damage.

2. Healthcare Clinical Research

In a healthcare setting, a hospital builds a knowledge graph to link patients, diseases, treatments, and clinical trials:

  1. A physician searches for:
    “Recommended treatment plans for Type 2 Diabetes patients with chronic kidney issues.”
  2. The knowledge graph references:
    • Patient histories detailing comorbidities, test results, and prescribed medications.
    • Clinical guidelines highlighting approved treatments for combined diabetes and kidney conditions.
    • Relevant clinical trials that are recruiting patients with both Type 2 Diabetes and chronic kidney disease.
  3. By integrating medical codes, research studies, and patient data into a unified graph, the physician is shown a comprehensive set of recommendations aligned with the latest guidelines, improving patient outcomes.

Implementing Knowledge Graphs on Azure

To extend the wind turbine and healthcare scenarios into a fully functioning knowledge graph environment, here’s how you can leverage Azure services—particularly Azure Cosmos DB—for a scalable, high-performance graph solution. The process below includes example Gremlin queries relevant to the GE Turbine Model X-200 use case.

1. Data Ingestion and Querying

Below is an example of how to populate your knowledge graph with data related to turbines and vibration issues:

    # Adding turbines (vertices)
    g.addV('turbine').property('id', 'X-200').property('model', 'GE Turbine Model X-200') \
                    .property('location', 'Texas wind farm').property('installed_year', 2019)
    
    g.addV('turbine').property('id', 'X-201').property('model', 'GE Turbine Model X-201') \
                    .property('location', 'Texas wind farm').property('installed_year', 2020)
    
    # Adding an event vertex for abnormal vibration
    g.addV('event').property('id', 'vibration-abnormal').property('type', 'vibration issue') \
                   .property('severity', 'high')
    
    # Creating relationships
    g.V('X-200').addE('has_issue').to(g.V('vibration-abnormal'))
    g.V('vibration-abnormal').addE('requires').to(
        g.addV('process').property('name', 'Maintenance')
    )

In a more automated scenario, these inserts could come from Azure Data Factory pipelines that parse IoT sensor data, maintenance logs, or unstructured text using Azure Cognitive Services.

Let’s try to query the graph and retrieve information using Gremlin traversals. For instance, to find all turbines with abnormal vibration:

    results = (
        g.V().hasLabel('turbine')
        .outE('has_issue')
        .inV().has('type', 'vibration issue')
        .values('id')
        .toList()
    )
    
    print("Turbines with abnormal vibration issues:", results)

Or query specific relationships, such as discovering which maintenance processes are required for a given turbine:

    maintenance_for_turbine = (
        g.V('X-200')
        .out('has_issue')
        .out('requires')
        .values('name')
        .toList()
    )
    
    print(f"Maintenance needed for Turbine X-200: {maintenance_for_turbine}")

2. Integration with Azure AI Services

Azure AI Search

  • Index your graph data using Azure AI Search to provide semantic and natural language query capabilities.
  • Implement semantic search to refine results, e.g., “Find turbines installed in 2019 that exhibit high vibration severity”.

Azure Cognitive Services

  • Text Analytics: Extract entities (e.g., “GE Turbine Model X-200,” “abnormal vibration”) from maintenance logs. - Custom Vision: Recognize objects in images (e.g., damaged rotor blades), then add them as nodes in your knowledge graph for visual context.

3. Graph RAG Implementation

A Retrieval-Augmented Generation (RAG) system combines vector-based retrieval with graph-based retrieval (structured knowledge):

  1. Ingest and index relevant documentation and sensor data into Azure AI Search (vector-based).
  2. Extract key entities and relationships with Gremlin in Azure Cosmos DB (graph-based).
  3. Hybrid approach at query time:
    • Vector retrieval for relevant text passages (manuals, logs).
    • Graph traversal to fetch connected data (e.g., turbines with similar issues).
  4. Synthesize a final answer using Azure OpenAI Service that incorporates both structured and unstructured sources.

4. Visualization and Analysis

  • Power BI or Azure Synapse Analytics: Visualize relationships (clusters of turbines exhibiting similar issues over time).
  • Azure Machine Learning: Integrate with the knowledge graph to build predictive models for forecasting failure risks based on multi-hop connections (location, vibration severity, installation date).

Example Queries for the Turbine Knowledge Graph

  1. Proactive Maintenance

    # Identify turbines installed before 2020 that have a vibration issue
from gremlin_python.process.traversal import P

results = g.V().hasLabel('turbine') \
          .has('installed_year', P.lt(2020)) \
          .outE('has_issue').inV().has('type', 'vibration issue') \
          .values('id').toList()

print("Older turbines with abnormal vibration:", results)

  2. Cross-Referencing Weather Factors

    # Suppose you have a 'weather' vertex with edges to turbines, tracking storms or extreme conditions
weather_impact = g.V().hasLabel('weather').has('severity', 'high') \
                 .inE('affects').outV().hasLabel('turbine') \
                 .values('id').toList()

print("Turbines affected by severe weather:", weather_impact)

  3. Finding Similar Issues in Other Turbines

    # Compare turbines sharing an event label like 'vibration issue'
similar_issues = g.V('X-200').out('has_issue').in('has_issue').hasLabel('turbine') \
                  .values('id').toList()
print("Turbines with similar issues as X-200:", similar_issues)

By combining these queries with Azure Cognitive Services for data ingestion, AI-powered entity extraction, and visualization in Power BI or Azure Synapse, you create a holistic knowledge graph solution that addresses real-time asset management challenges.

Conclusion

By merging NER-driven knowledge graphs with semantic retrieval and relationship-aware analysis, industries ranging from energy to healthcare can transcend the limitations of keyword-based search. These approaches allow for:

  • Precise, context-rich answers to complex operational or medical queries.
  • Integration of siloed data sources under a unified structure.
  • Scalable, real-time updates ensuring insights stay current.
  • Enhanced AI systems, from predictive maintenance to treatment recommendations.

As operations grow in complexity, knowledge graphs become a foundational technology—improving search accuracy, fueling advanced analytics, and accelerating decision-making in a world where every minute of downtime matters, and in healthcare, every insight can save lives.

You May Also Enjoy