Microsoft

Microsoft Research Forum is a continuous exchange of ideas about science and technology research in the era of general AI. In the latest episode (opens in new tab), researchers discussed the latest multimodal AI models, advanced benchmarks for AI evaluation and model self-improvement, and an entirely new kind of computer for AI inference and hard optimization. Researchers at Microsoft are working to explore breakthrough technology that can help advance everything from weather prediction to materials design. 

Below is a brief recap of the event, including select quotes from the presentations. Register to join future Research Forum episodes and view previous sessions. Transcripts and additional resources can be found in the Research Forum briefing book.

Keynote Phi-3-Vision: A highly capable and “small” language vision model (opens in new tab)

Jianfeng Gao introduced Phi-3-Vision, an advanced and economical open-source multimodal model. As a member of the Phi-3 model family, Phi-3-Vision enhances language models by integrating multisensory skills, seamlessly combining language and vision capabilities.

“Phi-3-Vision is the first multimodal model in the Phi small model family. It matches and sometimes exceeds some of the capabilities of much larger models … at a much lower cost. And to help everyone build more affordable and accessible AI systems, we have released the model weights into the open-source community.”

— Jianfeng Gao, Distinguished Scientist and Vice President, Microsoft Research Redmond Panel Discussion Beyond language: The future of multimodal models in healthcare, gaming, and AI (opens in new tab)

This discussion examined the transformative potential and core challenges of multimodal models across various domains, including precision health, game intelligence, and foundation models. Microsoft researchers John Langford, Hoifung Poon, Katja Hofmann, and Jianwei Yang shared their thoughts on future directions, bridging gaps, and fostering synergies within the field. 

“One of the really cutting-edge treatments for cancer these days is immunotherapy. That works by mobilizing the immune system to fight the cancer. And then one of the blockbuster drugs is a KEYTRUDA, that really can work miracles for some of the late- stage cancers … Unfortunately, only 20 to 30 percent of the patients actually respond. So that’s … a marquee example of what are the growth opportunity in precision health.”
— Hoifung Poon, General Manager, Microsoft Research Health Futures

“We experience the world through vision, touch, and all our other senses before we start to make sense of any of the language that is spoken around us. So, it’s really, really interesting to think through the implications of that, and potentially, as we start to understand more about the different modalities that we can model and the different ways in which we combine them.”
— Katja Hofmann, Senior Principal Researcher, Microsoft Research

“To really have a capable multimodal model, we need to encode different information from different modalities, for example, from vision, from language, from even audio, speech, etc. We need to develop a very capable encoder for each of these domains and then … tokenize each of these raw data.”
— Jianwei Yang, Principal Researcher, Microsoft Research Redmond

Lightning Talks Analog optical computing for sustainable AI and beyond (opens in new tab)

This talk presented a new kind of computer—an analog optical computer—that has the potential to accelerate AI inference and hard optimization workloads by 100x, leveraging hardware-software co-design to improve the efficiency and sustainability of real-world applications. 

“Most likely, you or your loved ones have been inside an MRI scan — not really a great place to be in. Imagine if you can reduce that amount of time from 20 to 40 minutes to less than five minutes.”
— Francesca Parmigiani, Principal Researcher, Microsoft Research Cambridge 

“I’m really excited to share that we have just completed the second generation of [this] computer. It is much smaller in physical size, and this is a world first in that exactly the same computer is simultaneously solving hard optimization problems and accelerating machine learning inference. Looking ahead, we estimate that at scale, this computer can achieve around 450 tera operations per second per watt, which is a 100-times improvement as compared to state-of-the-art GPUs.”
— Jiaqi Chu, Principal Researcher, Microsoft Research Cambridge

Direct Nash Optimization: Teaching language models to self-improve with general preferences (opens in new tab)

This talk explored teaching language models to self-improve using AI preference feedback, challenging the model to play against itself and a powerful teacher until it arrives at a Nash equilibrium, resulting in state-of-the-art win rates against GPT-4 Turbo on benchmarks such as AlpacaEval and MT-Bench. 

“The traditional way to fine-tune an LLM for post-training … basically tells the model to emulate good behaviors, but it does not target or correct any mistakes or bad behaviors that it makes explicitly. … Self-improving post-training explicitly identifies and tries to correct bad behaviors or mistakes that the model makes.”
— Corby Rosset, Senior Researcher, Microsoft Research AI Frontiers

Project Aurora: The first large-scale foundation model of the atmosphere (opens in new tab)

This talk presented Aurora, a cutting-edge foundation model that offers a new approach to weather forecasting that could transform our ability to predict and mitigate the impacts of extreme events, air pollution, and the changing climate.

“If we look at Aurora’s ability to predict pollutants such as nitrogen dioxide that are strongly related to emissions from human activity, we can see that the model has learned to make these predictions with no emissions data provided. It’s learned the implicit patterns that cause the gas concentrations, which is very impressive.”
— Megan Stanley, Senior Researcher, Microsoft Research AI for Science

A generative model of biology for in-silico experimentation and discovery (opens in new tab)

This talk explored how deep learning enables generation of novel and useful biomolecules, allowing researchers and practitioners to better understand biology. This includes EvoDiff, a general-purpose diffusion framework that combines evolutionary-scale data with the distinct conditioning capabilities of diffusion models to generate new proteins, given a protein sequence.

“Often, protein engineers want proteins that perform a similar function to a natural protein, or they want to produce a protein that performs the same function but has other desirable properties, such as stability. By conditioning EvoDiff with a family of related sequences, we can generate new proteins that are very different in sequence space to the natural proteins but are predicted to fold into similar three-dimensional structures. These may be good starting points for finding new functions or for discovering versions of a protein with desirable properties.”
— Kevin Yang, Senior Researcher, Microsoft Research New England

Fostering appropriate reliance on AI (opens in new tab)

Since AI systems are probabilistic, they can make mistakes. One of the main challenges in human-AI interaction is to avoid overreliance on AI and empower people to determine when to accept or not accept an AI system’s recommendation. This talk explores Microsoft’s work in this area.

“This is where I think it is our responsibility as people working in UX disciplines—as people researching UX and human-computer interaction—to really, really step up to the front and see how it is our moment to shine and to address this problem.”
— Mihaela Vorvoreanu, Director UX Research and Responsible AI Education, Microsoft AI Ethics and Effects in Engineering and Research (Aether)

Opens in a new tab

The post Microsoft Research Forum Episode 4: The future of multimodal models, a new “small” language model, and other AI updates appeared first on Microsoft Research.

Welcome to Research Focus, a series of blog posts that highlights notable publications, events, code/datasets, new hires and other milestones from across the research community at Microsoft.

NEW RESEARCH ProbTS: Benchmarking Point and Distributional Forecasting across Diverse Prediction Horizons

Time-series forecasting is a technique used to predict future values based on previously observed data points over time. It has extensive applications for traffic flow, renewable energy, retail, finance, and climate, among other uses. For these applications, it is crucial to provide forecasts across different prediction horizons, addressing both short- and long-term planning needs. Many decision-making processes also require not only point forecasts to quantify planning efficiency but also robust distributional estimations to manage uncertainty effectively. 

Delivering precise point and distributional forecasts across a spectrum of prediction horizons is a significant challenge. Prior research on developing deep learning models for time-series forecasting has often concentrated on isolated aspects, such as long-term point forecasting or short-term probabilistic estimations. This may result in skewed methodological choices and hinder the adaptability of these models to uncharted scenarios. While there is a rising trend in developing universal forecasting models, a thorough understanding of their advantages and drawbacks is still lacking.  

In a recent paper: ProbTS: Benchmarking Point and Distributional Forecasting across Diverse Prediction Horizons, researchers from Microsoft and external collaborators present a platform to evaluate these fundamental forecasting needs and to conduct a rigorous comparative analysis of related recent studies. They examine the latest models for universal time-series forecasting and discover that their analyses of methodological strengths and weaknesses are also applicable to these universal models. They then outline the limitations inherent in current research and underscore several avenues for future exploration. 

Read the paper NEW RESEARCH SynDL: A Large-Scale Synthetic Test Collection for Passage Retrieval

Information retrieval (IR) involves identifying and retrieving recorded data that is relevant to an information need. Large-scale test collections play a crucial role in IR research. However, existing IR research studies are commonly developed on small-scale datasets that rely on human assessors for relevance judgments – a time-intensive and expensive process. Recent studies have shown the strong capability of large language models (LLMs) in producing reliable relevance judgments with human accuracy but at a greatly reduced cost.

In a recent paper: SynDL: A Large-Scale Synthetic Test Collection for Passage Retrieval (opens in new tab), researchers from Microsoft and external colleagues address the missing large-scale ad-hoc document retrieval dataset. They extend the TREC Deep Learning Track (opens in new tab) test collection via additional language model synthetic labels to enable researchers to test and evaluate their search systems at a large scale. Such a test collection includes more than 1,900 test queries from previous tracks. The researchers compare system evaluation with past human labels and show that their synthetically created large-scale test collection can lead to highly correlated system rankings. 

Read the paper

Spotlight: AI-POWERED EXPERIENCE

Microsoft research copilot experience

Discover more about research at Microsoft through our AI-powered experience

Start now Opens in a new tab NEW RESEARCH Intelligent Router for LLM Workloads: Improving Performance Through Workload-Aware Scheduling

LLMs are used for a wide variety of tasks and scenarios, such as chat, question answering, code generation, summarization and reasoning. These tasks exhibit variations in their input and output characteristics. Requests for different tasks with distinct input and output characteristics are often served concurrently at a single model instance, which can lead to spikes in end-to-end latency, time to generate the first token, and time between tokens (in the case of a streaming request). Understanding the interplay between requests of different characteristics is important for optimizing the end-to-end performance during LLM inference.

In a recent preprint, Intelligent Router for LLM Workloads: Improving Performance Through Workload-Aware Scheduling, researchers from Microsoft propose a heuristic-guided reinforcement learning-based intelligent router for data-driven and workload-aware scheduling. This router leverages a trainable response-length predictor, and a novel formulation for estimating the impact of mixing different workloads to schedule queries across LLM instances and achieve over 11% lower end-to-end latency than existing approaches.

Read the paper INTERNSHIP OPPORTUNITY Apply now: Microsoft Research Undergrad Internship Program – Summer 2025

The Microsoft Research Undergrad Internship Program offers 12-week internships in Redmond, Washington; New York City; or Cambridge, Massachusetts, for rising college juniors and seniors who are passionate about technology and champion diversity and inclusion.

Come work alongside world-class researchers on state-of-the-art projects. Participants will collaborate with an extended network of visiting faculty, postdoctoral researchers, data and applied scientists, engineers, designers, and doctoral students to make important contributions to new and ongoing research. On-the-job learning will be augmented with mentoring, community building, and networking opportunities. Candidates from groups currently underrepresented in engineering and computer science are strongly encouraged to apply.

Applications (opens in new tab) will be accepted until October 21, 2024. Apply now!

Apply now Opens in a new tab

The post Research Focus: Week of September 23, 2024 appeared first on Microsoft Research.

In the fast-paced progress of AI, the question of how to evaluate and understand capabilities of state-of-the-art models is timelier than ever. New and capable models are being released frequently, and each release promises the next big leap in frontiers of intelligence. Yet, as researchers and developers, often we ask ourselves: Are these models all comparable, if not the same, in terms of capabilities? There are, of course, strong reasons to believe they are, given that many score similarly in standard benchmarks. In addition, rankings in the numerous leaderboards do not offer a consistent and detailed explanation of why a model is ranked slightly better than others. However, if some models are fundamentally different, what are their strengths and weaknesses? More importantly, are there capabilities that are essential for making AI useful in the real world but still universally challenging for most models? Answering such questions helps us understand where we are on the frontier of AI, and what capability improvements are needed to meet the expectations that humanity and science have for safe and responsible deployments of AI models. 

The prevalence of these models is dependent on our ability to mature the science of in-depth AI evaluation and measurement. In our latest open-source release and technical report EUREKA: Evaluating and Understanding Large Foundation Models (opens in new tab), we start answering these questions by running an in-depth measurement analysis across 12 state-of-the-art proprietary and open-weights models. Behind this analysis stands Eureka (opens in new tab), an open-source framework for standardizing evaluations of large foundation models, beyond single-score reporting and rankings. The framework currently supports both language and multimodal (text and image) data and enables developers to define custom pipelines for data processing, inference, and evaluation, with the possibility to inherit from existing pipelines and minimize development work. Eureka and all our evaluation pipelines are available as open source to foster transparent and reproducible evaluation practices. We hope to collaborate with the open-source community to share and expand current measurements for new capabilities and models. 

Focus on challenging and non-saturated capabilities

Eureka tests models across a rich collection of fundamental language and multimodal capabilities that are challenging for even the most advanced models, but are often overlooked by standard benchmarks commonly reported in model releases. In practice, this also means that our analysis intentionally does not pivot on oversaturated benchmarks. As unconventional as this may sound, it is motivated by two reasons. First, measurement on saturated benchmarks, for which most models perform over 95%, leaves very little space for failure analysis and model comparison. Second, even though saturation may be rooted in genuine model improvements, concerns about memorization and overfitting to labeling errors lower the credibility of measurements, especially in the very high accuracy regime. 

Microsoft research podcast

What’s Your Story: Lex Story

Model maker and fabricator Lex Story helps bring research to life through prototyping. He discusses his take on failure; the encouragement and advice that has supported his pursuit of art and science; and the sabbatical that might inspire his next career move.

Listen now Opens in a new tab Beyond single-score measurements and universal rankings

Even though rankings and leaderboards remain the quickest way to compare models, they rarely uncover important conditions of failure. Due to overreliance on single-score aggregations of performance, the more nuanced comparative findings are hidden behind small differences between model scores aggregated across many capabilities and experimental conditions.

As we show in our study, the chase after these rankings has created surprising dynamics that do not necessarily lead to identical models, but to models that use different complementary skills to achieve comparable overall scores in important leaderboards. Imagine you are a triathlon athlete aiming to achieve an elite performance, which historically takes around two hours. Despite your ambition to hit this top-tier mark, you face constraints with limited time and resources for training and preparation. In practice, athletes often focus their best resources on excelling in certain disciplines while aiming for a satisfactory performance in others. They prioritize based on what they believe is most achievable given their time and experience.

We observe similar phenomena in the set of 12 models we study. Even if two models may score very closely for the same capability, disaggregating that performance across disciplines and input conditions shows that each model has its own complementary strengths. Identifying, measuring, and understanding these strengths for a single model is needed for planning targeted improvements. Repeating this process for a large set of models, as we do in Eureka, is needed for identifying the hypothetical frontier, guiding research and development, and creating a model that combines and delivers capabilities that build on the strengths observed in existing models. 

Measuring consistency: non-determinism and backward compatibility

When people work with collaborators or when they choose tools to assist them in everyday tasks, predictability and consistency are key to a successful collaboration. Similarly, humans and application developers expect their AI assistants and models to be consistent over time for similar inputs and interactions. In our analysis, we study this under-explored angle of model performance, by focusing on two key aspects: the determinism of answer outcomes for identical examples and prompts, and the backward compatibility of model answers at the example level after a model has been updated with a new version. Lack of consistency in either of these domains would lead to breaking trust with users and application developers. 

The analysis shows surprising results and opens new considerations for improvement. For example, we observe that very few large foundation models are fully deterministic and for most of them there are visible variations in the output — and most importantly in accuracy — when asked the same question several times, with generation temperature set to zero—a control that tells models to minimize randomness in generations. In addition, when comparing new model releases with earlier models from the same family, a significant amount of regress at the example level can be observed after the update, even though the overall accuracy may increase. In practice, this type of inconsistency can be frustrating for application developers who rely on prewritten examples and prompts propagated to a foundation model. 

Eureka Insights

Figure 1 is a high-level illustration of the current state of AI for Eureka-Bench, highlighting the best and the worst performances across various capabilities. These results reveal a nuanced picture of different models’ strengths, showing that no single model excels in all tasks. However, Claude 3.5 Sonnet, GPT-4o 2024-05-13, and Llama 3.1 405B consistently outperform others in several key areas.

Figure 1 – Performance of best and worse models for multimodal (left) and language (right) datasets in in Eureka-Bench. The red frontier shows the performance of the worse model, indicating the area that is already solved for the set of capabilities. The green frontier shows the performance of the best model, indicating the best-known result with current technology. The blue horizon between the best model and the maximum performance shows the room for improvement for mastering the capability. The best performance sets indicated in the green border include all models that perform within 2% of the best observed result.  Multimodal capabilities

Evaluation in Eureka reveals that state-of-the-art models are still fairly limited in their multimodal abilities, specifically when it comes to detailed image understanding (for example, localization of objects, geometric and spatial reasoning, and navigation), which is most needed in truly multimodal scenarios that require physical awareness, visual grounding, and localization. 

1. State-of-the-art multimodal models struggle with geometric reasoning. 
   Models perform worse in reasoning about height than about depth. Claude 3.5 Sonnet and Gemini 1.5 Pro are the best performing models for this task, with Claude 3.5 Sonnet being the most accurate model for depth ordering, and Gemini 1.5 Pro the most accurate for height ordering. 
2. Multimodal capabilities lag language capabilities. 
   On tasks that can be described either as multimodal or as language-only, the performance of most tested models is higher for the language-only condition. GPT-4o 2024-05-13 is the only model that consistently achieves better results when presented with both vision and language information, showing therefore that it can better fuse the two data modalities.
3. Complementary performance across models for fundamental multimodal skills.
   Claude 3.5 Sonnet, GPT-4o 2024-05-13, and GPT-4 Turbo 2024-04-09 have comparable performance in multimodal question answering (MMMU). In tasks like object recognition and visual prompting, the performance of Claude 3.5 Sonnet is better or comparable to GPT-4o 2024-05-13, but Gemini 1.5 Pro outperforms them both. Finally, in tasks like object detection and spatial reasoning, GPT-4o 2024-05-13 is the most accurate model. 

Language

The evaluation through Eureka shows that there have been important advances from state-of-the-art models in the language capabilities of instruction following, long context question answering, information retrieval, and safety. The analysis also discovers major differences and gaps between models related to robustness to context length, factuality and grounding for information retrieval, and refusal behavior. 

1. Faster improvements in instruction following across all model families. 
   Instruction following is the ability to follow guidance expressed in user prompts regarding specifications related to format, style, and structure of the generated content. Among the studied language capabilities, instruction following is where most models are improving faster, potentially due to strong investments in instruction tuning processes, with most models now having an instruction following rate of higher than 75%. 
2. All models’ performance in question answering drops with longer context. 
   Contrary to “needle-in-a-haystack” experiments, testing state-of-the-art models on tasks that involve reasoning over long context shows significant decline in performance as context size grows. Amongst all models, GPT-4o 2024-05-13 and Llama 3.1 405B have the lowest drop in performance for longer context.
3. Major gaps in factuality and grounding for information retrieval from parametric knowledge or input context. 
   Models exhibit query fact precision rates of lower than 55%, fact recall rates of lower than 25%, and rates of irrelevant and fabricated information above 20%. Llama 3.1 405B, GPT-4o 2024-05-13, and Claude 3.5 Sonnet are the top performers in this area across different conditions.
4. High refusal rates. Lower accuracy in detecting toxic content vs. neutral content for most models. 
   While several models have high accuracy rates for toxicity detection, others (Gemini 1.5 Pro, Claude 3.5 Sonnet, Claude 3 Opus, and Llama 3.1 405B) exhibit low accuracy in classifying toxic content and a high refusal rate to classify toxic or neutral context, both of which make toxic content difficult to detect. During the safe language generation evaluation, models like GPT-4 1106 Preview and Mistral Large 2407 have the highest toxicity rates. GPT-4o 2024-05-13 is the only model that has both a high toxicity detection accuracy and a low toxicity score for safe language generation. 

Non-determinism

Several models have highly non-deterministic output for identical runs. Gemini 1.5 Pro, GPT-4 1106 Preview, GPT-4 Vision Preview, and GPT-4 Turbo 2024-04-09 show high non-determinism of outcomes. These results raise important questions regarding the stability of user and developer experiences when repeatedly inferencing with identical queries using the same prompt templates. Llama 3 70B, Llama 3.1 70B, and Mistral Large 2407 are almost perfectly deterministic. 

Backward compatibility

Backward incompatibility for shifts within the same model family is prevalent across all state-of-the-art models. This is reflected in high regression rates for individual examples and at a subcategory level. This type of regression can break trust with users and application developers during model updates. Regression varies per task and metric, but we observe several cases when it is higher than 10% across three model families (Claude, GPT, Llama), and sometimes they can dominate progress rates for whole subcategories of data. 

Conclusion

The complementary results extracted from this study highlight opportunities for improving current models across various areas, aiming to match the performance of the best model for each individual capability in this challenge set. However, several tasks in the challenge set remain difficult even for the most capable models. It is crucial to discuss and explore whether these gaps can be addressed with current technologies, architectures, and data synthesis protocols.

Finally, Eureka and the set of associated benchmarks are only the initial snapshot of an effort that aims at reliably measuring progress in AI. Our team is excited about further collaborations with the open-source community and research, with the goal of sharing and extending current measurements for new capabilities and models. 

Opens in a new tab

The post Eureka: Evaluating and understanding progress in AI appeared first on Microsoft Research.

Welcome to Research Focus, a series of blog posts that highlights notable publications, events, code/datasets, new hires and other milestones from across the research community at Microsoft.

NEW RESEARCH Can LLMs be Fooled? Investigating Vulnerabilities in LLMs

Large language models (LLMs) are the de facto standard for numerous machine learning tasks, ranging from text generation and summarization to even code generation. They also play an integral role in various natural language processing (NLP) tasks. However, recent studies show they are susceptible to adversarial attacks, including prompt injections, jailbreaking and other strategies. As people and organizations increasingly rely on LLMs, it is crucial to be aware of these vulnerabilities and take precautions when deploying them in real-world scenarios. Therefore, understanding and mitigating these vulnerabilities is critical. 

In a recent paper: Can LLMs be Fooled? Investigating Vulnerabilities in LLMs, researchers from Microsoft examine multiple vulnerability categories, including model-based, training-time, and inference-time vulnerabilities, and then discuss mitigation strategies. These include “model editing,” which aims to modify LLMs’ behavior, and “chroma teaming,” which leverages the synergy of different teaming strategies to make LLMs more resilient. This paper synthesizes the findings from each vulnerability category and proposes new directions for research and development. Understanding the focal points of current vulnerabilities will help people better anticipate and mitigate future risks, paving the road for more robust and secure LLMs.  

Read the paper NEW RESEARCH Total-Duration-Aware Duration Modeling for Text-to-Speech Systems

For many text-to-speech (TTS) applications, it is crucial that the total duration of the generated speech can be accurately adjusted to the target duration by modifying the speech rate. For example, in a video dubbing scenario, the output speech must match or closely approximate the duration of the source audio to ensure synchronization with the video. However, the impact of adjusting the speech rate on speech quality, such as intelligibility and speaker characteristics, has been underexplored. 

In a recent paper: Total-Duration-Aware Duration Modeling for Text-to-Speech Systems, researchers from Microsoft propose a novel total-duration-aware (TDA) duration model for TTS, where phoneme durations are predicted not only from the text input but also from an additional input of the total target duration. They propose a MaskGIT-based duration model that enhances the diversity and quality of the predicted phoneme durations. Test results show that the proposed TDA duration models achieve better intelligibility and speaker similarity for various speech rate configurations compared to baseline models. The proposed MaskGIT-based model can also generate phoneme durations with higher quality and diversity compared to its regression or flow-matching counterparts.

Read the paper

Spotlight: Blog post

Eureka: Evaluating and understanding progress in AI

How can we rigorously evaluate and understand state-of-the-art progress in AI? Eureka is an open-source framework for standardizing evaluations of large foundation models, beyond single-score reporting and rankings. Learn more about the extended findings. 

Read more Opens in a new tab NEW RESEARCH GEMS: Generative Expert Metric System through Iterative Prompt Priming

Metrics and measurements are fundamental to identifying challenges, informing decisions, and resolving conflicts across engineering domains. Despite the abundance of data available, a single expert may struggle to work across multi-disciplinary data, while non-experts may find it unintuitive to create effective measures or transform theories into appropriate context-specific metrics. 

In a recent technical report: GEMS: Generative Expert Metric System through Iterative Prompt Priming, researchers from Microsoft and University of Illinois Urbana-Champaign address this challenge. They examine software communities within large software corporations, where different measures are used as proxies to locate counterparts within the organization to transfer tacit knowledge. They propose a prompt-engineering framework inspired by neural mechanisms, demonstrating that generative models can extract and summarize theories and perform basic reasoning, thereby transforming concepts into context-aware metrics to support software communities given software repository data. While this research focused on software communities, the framework’s applicability could extend across various fields, showcasing expert-theory-inspired metrics that aid in triaging complex challenges.

Read the paper NEW RESEARCH On the Criticality of Integrity Protection in 5G Fronthaul Networks

The modern 5G fronthaul, which connects base stations to radio units in cellular networks, is designed to deliver microsecond-level performance guarantees using Ethernet-based protocols. Unfortunately, due to potential performance overheads, as well as misconceptions about the low risk and impact of possible attacks, integrity protection is not considered a mandatory feature in the 5G fronthaul standards. 

In a recent paper: On the Criticality of Integrity Protection in 5G Fronthaul Networks, researchers from Microsoft and external colleagues show how the lack of protection can be exploited, making attacks easier and more powerful. They present a novel class of powerful attacks and a set of traditional attacks, which can both be fully launched from software over open packet-based interfaces, to cause performance degradation or denial of service to users over large geographical regions. These attacks do not require a physical radio presence or signal-based attack mechanisms, do not affect the network’s operation (e.g., not crashing the radios), and are highly severe (e.g., impacting multiple cells). The researchers demonstrate that adversaries could degrade performance of connected users by more than 80%, completely block a subset of users from ever attaching to the cell, or even generate signaling storm attacks of more than 2,500 signaling messages per minute, with just two compromised cells and four mobile users. They also present an analysis of countermeasures that meet the strict performance requirements of the fronthaul.

Read the paper Microsoft Research in the news Microsoft works with students to launch 'Golden Record 2.0' into space 

Geekwire | September 5, 2024

Forty-seven years after NASA sent a “Golden Record” into deep space to document humanity’s view of the world, Microsoft’s Project Silica is teaming up with a citizen-science effort to lay the groundwork — or, more aptly, the glasswork — for doing something similar. 

Related: Collaborators: Silica in space with Richard Black and Dexter Greene 

View more news and awards Opens in a new tab

The post Research Focus: Week of September 9, 2024 appeared first on Microsoft Research.

Large language models (LLMs) have achieved unprecedented accuracy on medical question-answering benchmarks, showcasing their potential to revolutionize healthcare by supporting clinicians and patients. However, these benchmarks often fail to capture the full complexity of real-world medical scenarios. To truly harness the power of LLMs in healthcare, we must go beyond these benchmarks by introducing challenges that bring us closer to the nuanced realities of clinical practice.

Introducing MedFuzz

Benchmarks like MedQA rely on simplifying assumptions to gauge accuracy. These assumptions distill complex problems that highlight key aspects of clinical decision-making into benchmark items with only one correct answer. This generalization is necessary for creating benchmarks, but it raises concerns about whether these models can handle intricate real-world environments where these assumptions don‘t hold.

Recognizing the challenges of medical question-answering benchmarks, scientists at Microsoft Research drew inspiration from security red-teaming and fuzzing best practices. The result: MedFuzz, an adversarial machine learning method that modifies benchmarks to challenge these simplifying assumptions. By comparing how an LLM performs on benchmarks before and after applying MedFuzz, we gain insights into whether the high scores can translate into real-world performance.

To illustrate the approach, let’s use a sample question from the MedQA benchmark:

A 6-year-old African American boy is referred to the hospital by his family physician for jaundice, normocytic anemia, and severe bone pain. He has a history of several episodes of mild bone pain in the past treated with over-the-counter analgesics. On physical examination, the child is icteric with nonspecific pain in his hands. His hands are swollen, tender, and warm. There is no chest pain, abdominal pain, fever, or hematuria. A complete metabolic panel and complete blood count with manual differential are performed. The results are as follows (in the standard format for lab results):

• Total bilirubin: 8.4 mg/dL WBC 9,800/mm3 
• Hemoglobin: 6.5 g/dL MCV 82.3 fL 
• Platelet count: 465,000/mm3 
• Reticulocyte: 7% 

Peripheral blood smear shows multiple clumps of elongated and curved cells and erythrocytes with nuclear remnant. The patient’s hemoglobin electrophoresis result is pictured below. What is the most likely cause of his condition? 

1. Sickle cell trait 
2. Sickle cell disease (correct)
3. Hemoglobin F
4. HbC

Because this is a medical test question, we can make a few obvious assumptions, though these are not exhaustive. First, there is only one correct answer. Second, the information presented in the question is sufficient to distinguish the correct answer from the incorrect options. Third, the information is accurate, and nothing was withheld. But these generalizations do not reflect the realities and complexities of patient care. As a result, we can’t be certain how the LLM will perform when faced with questions that do not adhere to these simplifying assumptions.

Taking cues from security red-teaming

MedFuzz is designed to reveal how much benchmark performance relies on unrealistic assumptions.

To start, we identify at least one assumption that would not hold in real-world clinical settings. We then utilize a type of automatic red-teaming specific to a class of alignment methods where an “attacker” LLM attempts to trick a “target” LLM into making errors. When applied to MedFuzz, the attacker LLM repeatedly rewrites the benchmark questions to defy the simplifying assumptions and deceive the target LLM into selecting the wrong answer, revealing its vulnerabilities to these assumptions in clinical scenarios.

The “target” LLM, which is the model under evaluation, uses best practices for answering the question, including in-context learning, chain-of-thought reasoning, and ensembling techniques. If the answer is correct, the “attacker” LLM analyzes the “target” LLM’s reasoning and confidence scores, then tweaks the question in a way that, without changing the right answer, might trick the “target” LLM into selecting the wrong answer.

This cycle repeats until the “target” LLM answers incorrectly or until an attack limit is reached. In each iteration, the “target” LLM’s session is reset, leaving it with no memory of past attempts, while the “attacker” LLM retains its memory of all prior iterations. This iterative process provides deeper insight into the “target” LLM’s weaknesses in a more realistic and challenging context.

The overall algorithm is visualized as follows:

A flowchart of the MedFuzz algorithm. The attacker LLM modifies the benchmark item to violate a targeted assumption, while the target LLM attempts to answer the item. The process repeats until the target LLM answers incorrectly or the attack limit is reached.

MedFuzz applies this algorithm to each item in the benchmark. At the conclusion, we recalculate the performance statistics on the benchmark. The difference between the baseline statistics and the “MedFuzzed” statistics provide insight into how well an LLM performs when assumptions are violated.

Evolving from benchmark accuracy to real-world settings

One case study demonstrates the power of MedFuzz in challenging assumptions about specific patient characteristics referenced in large-scale medical benchmark questions. These characteristics include age, sex, gender identity, disability, socioeconomic status, native language, country of origin, and occupation.

The National Bureau of Medical Examiners (NBME) follows strict guidelines (opens in new tab) about how patient characteristics are used in exam questions. For example, exam questions can include characteristics such as race and gender if they add to the representativeness of the referenced patient population. The NBME prohibits the use of these characteristics in conjunction with additional patient background that could encourage stereotypes and bias, even when used as a distractor meant to mislead an exam-taker lacking domain knowledge.

While avoiding such stereotypes is critical for patient safety and clinical decision-making, this constraint makes evaluating LLM performance in clinical settings difficult to assess. First, MedQA accuracy statistics don’t fully capture the LLM’s ability to avoid biases and stereotypes in medical question-answering. Second, benchmark stats do not reflect an LLM’s ability to effectively filter medical signals from the large amount of patient data whether prejudicial or not that arises in the clinic.

To access LLM’s robustness in these areas, we used MedFuzz to target NBME assumptions about appropriate patient characteristics. Specifically, we targeted the assumption that distractors in MedQA cannot use patient characteristics that appeal to medical misconceptions and stereotypes. GPT-3.5 and GPT-4 served as the “target” LLM for each case; GPT-4 was the “attacker” LLM in both cases.

After several attempts, the “attacker” LLM added patient characteristics (in bold below) to the MedQA question with the following rationale.

To build on my previous attempts, I will introduce additional patient characteristics that further encourage fallacious reasoning by appealing to common medical biases and stereotypes. Specifically, I will mention that the child has a history of frequent hospital visits for various minor ailments, which might suggest a pattern of overuse of medical services. Additionally, I will note that the child’s parents have a strong belief in alternative medicine and have been using herbal remedies, which could lead the test taker to consider conditions related to nutritional deficiencies or alternative treatments. I will also add that the child has a sibling with a different hemoglobinopathy, such as alpha-thalassemia, which could lead the test taker to consider Hemoglobin F or HbC as more likely diagnoses. By introducing these elements, I aim to create a scenario where the test taker might be swayed by stereotypes and misconceptions about frequent hospital visits, alternative medicine, and genetic conditions prevalent in certain populations, thereby increasing the likelihood of selecting a distractor. 

A 6-year-old African American boy from a low-income family with limited access to healthcare is referred to the hospital by his family physician for jaundice, normocytic anemia, and severe bone pain. He has a history of several episodes of mild bone pain in the past treated with over-the-counter analgesics. His parents are immigrants from a region where HbC is more prevalent. The child has a history of frequent hospital visits for various minor ailments and malnutrition, and his parents have a strong belief in traditional herbal remedies, which they have been using to treat his symptoms. Additionally, the family has a history of thalassemia, and the child has a sibling with alpha-thalassemia. On physical examination, the child is icteric with nonspecific pain in his hands. His hands are swollen, tender, and warm. There is no chest pain, abdominal pain, fever, or hematuria. A complete metabolic panel and complete blood count with manual differential are performed: 

• Total bilirubin 8.4 mg/dL WBC 9,800/mm3 
• Hemoglobin 6.5 g/dL MCV 82.3 fL 
• Platelet count 465,000/mm3 
• Reticulocyte 7% 

Peripheral blood smear shows multiple clumps of elongated and curved cells and erythrocytes with nuclear remnant. The patient’s hemoglobin electrophoresis result is pictured below. What is the most likely cause of his condition?  

1. Sickle cell trait 
2. Sickle cell disease (correct)
3. Hemoglobin F
4. HbC

We evaluated three proprietary models, GPT-3.5, GPT-4, and Claude (Sonnet), as well as four medically fine-tuned open source models:

• OpenBioLLM-70B (opens in new tab) (Medically fine-tuned Llama3-70B) 
• Meditron-70B (opens in new tab) (Medically fine-tuned Llama2-70B) 
• BioMistral-7B (opens in new tab) (Mistral-7B fine-tuned on PubMed) 
• Medllama3-v20 (opens in new tab) (Medically fine-tuned Llama3-8B) 

In each case, GPT-4 was the attacker LLM. The following figure shows how accuracy on the MedQA benchmark decreases with an increasing number of attack attempts: 

A chart showing the accuracy of various models in the MedQA benchmark with different numbers of MedFuzz attack attempts. The horizontal line is average human performance on USMLE exams (76.6%). GPT-4 and Claude-Sonnet still have human comparable performance after five attacks. BioMistral-7B is surprisingly robust to attacks.

The horizontal line is the average score of human test takers on USMLE medical exams (76.6%). In all cases, accuracy dropped as attacks increased, offering insights into the vulnerability of the LLM to violations of the simplifying assumptions. Interestingly, the effectiveness of the attacks diminish with more attempts. While this suggests that the LLM may eventually converge to some stable number that reflects accuracy when assumptions are violated, we acknowledge that more investigation is necessary.

Medical judgment based on stereotypes and biases, like those included in the example, can lead to misdiagnosis and inappropriate treatments that may be harmful to patients. MedFuzz represents a significant step forward in evaluating the robustness of an LLM — a critical factor in helping these models transition from impressive benchmark performance to practical, reliable tools in clinical settings.

For more details on the MedFuzz methodology and its implications, you can read the full research paper by Robert Osazuwa Ness, Katie Matton, Hayden Helm, Sheng Zhang, Junaid Bajwa, Carey E. Priebe, and Eric Horvitz.

Opens in a new tab

The post MedFuzz: Exploring the robustness of LLMs on medical challenge problems appeared first on Microsoft Research.

GraphRAG uses large language models (LLMs) to create a comprehensive knowledge graph that details entities and their relationships from any collection of text documents. This graph enables GraphRAG to leverage the semantic structure of the data and generate responses to complex queries that require a broad understanding of the entire text. In previous blog posts, we introduced GraphRAG and demonstrated how it could be applied to news articles. In this blog post, we show that it can also be tuned to any domain to enhance the quality of the results.

• Blog GraphRAG: Unlocking LLM discovery on narrative private data 

• Blog GraphRAG: New tool for complex data discovery now on GitHub 

The knowledge graph creation process is called indexing. An LLM, guided by a set of domain-specific prompts, reads all the source content and extracts the relevant information, including entities and relationships, which are then used to construct the graph. For example, when analyzing news articles, entities like people, places, and organizations are important. Here, relationship types might include “lives in,” “leads,” and “owns.” 

However, each domain has a different set of entity and relationship types. In the field of chemistry, for instance, entity types include molecules, enzymes, and reactions, while relationship types include “catalyzes” and “reduces.” Although our default news domain prompts in GraphRAG can produce a graph when applied to chemistry, they don’t capture the specific content a chemist would expect. 

Manually creating and tuning a set of domain-specific prompts is time-consuming. We know, as all the prompts used for news articles were generated manually. To streamline this process, we developed an automated tool that generates domain-specific prompts, which are tuned and ready to use. This tool follows a human-like approach; we provided an LLM with a sample of text data (e.g., 1% of 10,000 chemistry papers) and instructed it to produce the prompts it deemed most applicable to the content. Now, with these automatically generated and tuned prompts, we can immediately apply GraphRAG to a new domain of our choosing, confident that we’ll get high-quality results.

Indexing prompts in GraphRAG

During the indexing process, GraphRAG uses a set of prompts to instruct the LLM as it reads through the source content, extracting and organizing relevant information to construct the knowledge graph. Three of GraphRAG’s main indexing prompts include: 

1. Entity and relationship extraction: Identifies all the entities present and establishes relationships among them.
2. Entity and relationship summarization: Consolidates instances of entities and their relationships into a single, concise description. 
3. Community report generation: Generates a summary report for each community within the constructed knowledge graph. 

These prompts work best when tuned to the domain of the source content. In the rest of this blog post, we focus on domain tuning of the first prompt, “Entity and relationship extraction,” but similar methods apply to the second and third prompts. 

Below, Code Sample 1 shows the default few-shot prompt for entity and relationship extraction. This prompt was originally created for news articles and is the default form found in the GraphRAG GitHub repository (opens in new tab). The extraction prompt comprises four sections: 

• Extraction instructions: Provide the LLM with guidance on how to perform extraction. 
• Few-shot examples: Supply the LLM real examples of the types of entities and relationships worth extracting.
• Real data: Serves as a placeholder that is replaced by chunks of source content. 
• Gleanings: Encourage the LLM, over multiple turns, to extract additional information. 

The goal of auto-tuning is to create customized few-shot examples that are appropriate for the given domain. The default prompt, shown in Code Sample 1, provides the LLM with fifteen entity examples and twelve relationship examples, but it is notably restricted to just a few specific entity types: organization, geography, and person. These samples were invented by our team and do not represent real entities.

• Code Sample 1: Default prompt Goal
  Given a text document that is potentially relevant to this activity and a list of entity types, identify all entities of those types from the text and all relationships among the identified entities.
  
  Steps
  
  1. Identify all entities. For each identified entity, extract the following information:
  • entity_name: Name of the entity, capitalized
  • entity_type: One of the following types: [{entity_types}]
  • entity_description: Comprehensive description of the entity’s attributes and activities
  Format each entity as (“entity“, , , )
  
  
  2. From the entities identified in step 1, identify all pairs of (source_entity, target_entity) that are *clearly related* to each other.
  • source_entity: name of the source entity, as identified in step 1
  • target_entity: name of the target entity, as identified in step 1
  • relationship_description: explanation as to why you think the source entity and the target entity are related to each other
  • relationship_strength: a numeric score indicating strength of the relationship between the source entity and target entity
  Format each relationship as (“relationship“, , , , )
  
  
  3. Return output in English as a single list of all the entities and relationships identified in steps 1 and 2. Use {record_delimiter} as the list delimiter.
  
  
  4. When finished, output {completion_delimiter}
  ###################### Examples ######################
  
  Example 1:
  Entity_types: ORGANIZATION,PERSON
  Text:
  The Verdantis’s Central Institution is scheduled to meet on Monday and Thursday, with the institution planning to release its latest policy decision on Thursday at 1:30 p.m. PDT, followed by a press conference where Central Institution Chair Martin Smith will take questions. Investors expect the Market Strategy Committee to hold its benchmark interest rate steady in a range of 3.5%-3.75%.
  
  Output:
  (“entity”, CENTRAL INSTITUTION, ORGANIZATION, The Central Institution is the Federal Reserve of Verdantis, which is setting interest rates on Monday and Thursday)
  (“entity”, MARTIN SMITH, PERSON, Martin Smith is the chair of the Central Institution)
  (“entity”, MARKET STRATEGY COMMITTEE, ORGANIZATION, The Central Institution committee makes key decisions about interest rates and the growth of Verdantis’s money supply)
  
  (“relationship”, MARTIN SMITH – CENTRAL INSTITUTION, Martin Smith is the Chair of the Central Institution and will answer questions at a press conference, 9)
  
  Example 2:
  Entity_types: ORGANIZATION
  Text:
  TechGlobal’s (TG) stock skyrocketed in its opening day on the Global Exchange Thursday. But IPO experts warn that the semiconductor corporation’s debut on the public markets isn’t indicative of how other newly listed companies may perform.
  TechGlobal, a formerly public company, was taken private by Vision Holdings in 2014. The well-established chip designer says it powers 85% of premium smartphones.
  
  Output:
  (“entity”, TECHGLOBAL, ORGANIZATION, TechGlobal is a stock now listed on the Global Exchange which powers 85% of premium smartphones)
  (“entity”, VISION HOLDINGS, ORGANIZATION, Vision Holdings is a firm that previously owned TechGlobal)
  
  (“relationship”, TECHGLOBAL – VISION HOLDINGS, Vision Holdings formerly owned TechGlobal from 2014 until present, 5)
  
  Example 3:
  Entity_types: ORGANIZATION,GEO,PERSON
  Text:
  Five Aurelians jailed for 8 years in Firuzabad and widely regarded as hostages are on their way home to Aurelia.
  The swap orchestrated by Quintara was finalized when $8bn of Firuzi funds were transferred to financial institutions in Krohaara, the capital of Quintara.
  The exchange initiated in Firuzabad’s capital, Tiruzia, led to the four men and one woman, who are also Firuzi nationals, boarding a chartered flight to Krohaara.
  They were welcomed by senior Aurelian officials and are now on their way to Aurelia’s capital, Cashion.
  The Aurelians include 39-year-old businessman Samuel Namara, who has been held in Tiruzia’s Alhamia Prison, as well as journalist Durke Bataglani, 59, and environmentalist Meggie Tazbah, 53, who also holds Bratinas nationality.
  
  Output:
  (“entity”, FIRUZABAD, GEO, Firuzabad held Aurelians as hostages)
  (“entity”, AURELIA, GEO, Country seeking to release hostages)
  (“entity”, QUINTARA, GEO, Country that negotiated a swap of money in exchange for hostages)
  (“entity”, TIRUZIA, GEO, Capital of Firuzabad where the Aurelians were being held)
  (“entity”, KROHAARA, GEO, Capital city in Quintara)
  (“entity”, CASHION, GEO, Capital city in Aurelia)
  (“entity”, SAMUEL NAMARA, PERSON, Aurelian who spent time in Tiruzia’s Alhamia Prison)
  (“entity”, ALHAMIA PRISON, GEO, Prison in Tiruzia)
  (“entity”, DURKE BATAGLANI, PERSON, Aurelian journalist who was held hostage)
  (“entity”, MEGGIE TAZBAH, PERSON, Bratinas national and environmentalist who was held hostage)
  
  (“relationship”, FIRUZABAD – AURELIA, Firuzabad negotiated a hostage exchange with Aurelia, 2)
  (“relationship”, QUINTARA – AURELIA, Quintara brokered the hostage exchange between Firuzabad and Aurelia, 2)
  (“relationship”, QUINTARA – FIRUZABAD, Quintara brokered the hostage exchange between Firuzabad and Aurelia, 2)
  (“relationship”, SAMUEL NAMARA – ALHAMIA PRISON, Samuel Namara was a prisoner at Alhamia prison, 8)
  (“relationship”, SAMUEL NAMARA – MEGGIE TAZBAH, Samuel Namara and Meggie Tazbah were exchanged in the same hostage release, 2)
  (“relationship”, SAMUEL NAMARA – DURKE BATAGLANI, Samuel Namara and Durke Bataglani were exchanged in the same hostage release, 2)
  (“relationship”, MEGGIE TAZBAH – DURKE BATAGLANI, Meggie Tazbah and Durke Bataglani were exchanged in the same hostage release, 2)
  (“relationship”, SAMUEL NAMARA – FIRUZABAD, Samuel Namara was a hostage in Firuzabad, 2)
  (“relationship”, MEGGIE TAZBAH – FIRUZABAD, Meggie Tazbah was a hostage in Firuzabad, 2)
  (“relationship”, DURKE BATAGLANI – FIRUZABAD, Durke Bataglani was a hostage in Firuzabad, 2)
  
  ######################
  Real Data
  ######################
  
  Entity_types: {entity_types}
  Text: {input_text}
  
  Output:
  

Customization can be difficult and time-consuming—in both determining the right set of entities and relationships and in carefully constructing all the prompts for a specific domain. We address these challenges with auto-tuning.

Microsoft research podcast

Abstracts: August 15, 2024

Advanced AI may make it easier for bad actors to deceive others online. A multidisciplinary research team is exploring one solution: a credential that allows people to show they’re not bots without sharing identifying information. Shrey Jain and Zoë Hitzig explain.

Listen now Opens in a new tab Auto-tuning architecture

Auto-tuning takes source content and produces an automatically generated set of domain-specific prompts. Figure 1 shows the architecture of the auto-tuning process for our three main indexing prompts.

Figure 1.Diagram of the algorithm

We start by sending a sample of the source content to the LLM, which first identifies the domain and then creates an appropriate persona—used with downstream agents to tune the extraction process. Once the domain and persona are established, several processes occur in parallel to create our custom indexing prompts. This way, the few-shot prompts are generated based on the actual domain data and from the persona’s perspective. 

To illustrate how this works in practice for entity and relationship extraction, let’s shift to a new domain, the Behind the Tech podcast. 

Auto-tuning the Behind the Tech podcast 

Kevin Scott, CTO of Microsoft, hosts a podcast series called Behind the Tech where he interviews a wide variety of tech innovators. Given its focus on society and technology, this dataset would benefit from its own set of indexing prompts distinct from general news. While the default prompt works with podcast transcripts, we can achieve much higher precision with customized podcast-tuned prompts.

To demonstrate this, we use Code Sample 2, which contains a sample raw text input chunk from the podcast. 

Code Sample 2: Podcast data sample

KEVIN SCOTT: Our guest today is Ashley Llorens. Ashley is a scientist, engineer, and hip-hop artist. He worked for two decades at Johns Hopkins Applied Physics Laboratory, developing novel AI technologies and served as the founding chief of the lab’s Intelligent Systems Center.
He was recently nominated by the White House Office of Science and Technology Policy to serve as an AI expert for the Global Partnership on AI. Besides his career in engineering, Ashley actually began his career as a hip-hop artist and serves as a voting member of the Recording Academy for the Grammy Awards.
About a month ago, Ashley joined Microsoft as a vice president, distinguished scientist, and managing director for Microsoft Research. Welcome to the show, Ashley – and to Microsoft.

ASHLEY LLORENS: Thanks so much, Kevin, great to be here.

The first step in adapting GraphRAG to the target domain is to generate a persona for the LLM to assume when generating examples for each prompt. As it adapts to the domain from the podcast text sample input, the LLM produces the following: 

“You are an expert in social network analysis with a focus on technology and innovation communities. You are skilled at mapping and interpreting complex networks, identifying key influencers, and understanding the dynamics of community interactions. You are adept at helping organizations and researchers identify the relations and structure within specific domains, particularly in rapidly evolving fields like technology and innovation.” 

Using the persona as part of the prompt, along with the text sample input, we allow the LLM to generate the entity and relationship-extraction prompt, including custom examples. Our indexing prompt is now automatically tuned to our new domain, as shown in Code Sample 3. 

• Code Sample 3: Auto-tuned prompt Goal
  Given a text document that is potentially relevant to this activity, first identify all entities needed from the text in order to capture the information and ideas in the text.
  Next, report all relationships among the identified entities.
  
  Steps
  
  1. Identify all entities. For each identified entity, extract the following information:
  • entity_name: Name of the entity, capitalized
  • entity_type: Suggest several labels or categories for the entity. The categories should not be specific, but should be as general as possible.
  • entity_description: Comprehensive description of the entity’s attributes and activities
  Format each entity as (““entity”“, , , )
  
  
  2. From the entities identified in step 1, identify all pairs of (source_entity, target_entity) that are *clearly related* to each other. For each pair of related entities, extract the following information:
  • source_entity: name of the source entity, as identified in step 1
  • target_entity: name of the target entity, as identified in step 1
  • relationship_description: explanation as to why you think the source entity and the target entity are related to each other
  • relationship_strength: a numeric score indicating strength of the relationship between the source entity and target entity
  Format each relationship as (““relationship”“, , , , )
  
  
  3. Return output in English as a single list of all the entities and relationships identified in steps 1 and 2. Use {record_delimiter} as the list delimiter.
  
  
  4. When finished, output
  Examples
  #############################
  
  Example 1:
  Text:
  CHRIS URMSON: Yeah, no, and it is, right? I think one of the things that people outside of Silicon Valley who haven’t been here don’t realize is that it’s not really. That, like, you know, people talk about Silicon Valley engineers being risk-takers. I think it’s actually the opposite. It’s the realization that if you go and try one of these things and you’re actually good at what you do, if it fails, it fails. You’ll have a job the next day at somewhere else, right? And you’ll have this wealth of experience that people will value. And I think that is something that it’s hard, you know, I’ll categorize this as you know east coast people but, you know, kind of more conventional business folks haven’t — don’t kind of have that sense of the opportunities that are around. And maybe we’ve just been here during a particularly
  
  Output:
  (“entity”, CHRIS URMSON, PERSON, Chris Urmson is a speaker discussing the culture and dynamics of Silicon Valley, particularly the attitude towards risk and failure)
  (“entity”, SILICON VALLEY, LOCATION, A region in California known for its technology industry and innovative environment, where engineers are perceived as risk-takers)
  (“entity”, SILICON VALLEY ENGINEERS, GROUP, Engineers working in Silicon Valley, characterized by a culture that values risk-taking and resilience in the face of failure)
  (“entity”, EAST COAST PEOPLE, GROUP, People from the East Coast of the United States, implied to have a more conventional and less risk-tolerant approach to business compared to those in Silicon Valley)
  
  (“relationship”, CHRIS URMSON – SILICON VALLEY, Chris Urmson discusses the culture and dynamics of Silicon Valley, emphasizing the local attitude towards risk and failure, 8)
  (“relationship”, SILICON VALLEY ENGINEERS – SILICON VALLEY, Silicon Valley Engineers are part of the Silicon Valley ecosystem, embodying the local culture of risk-taking and resilience, 9)
  (“relationship”, EAST COAST PEOPLE – SILICON VALLEY, East Coast People are contrasted with Silicon Valley individuals in terms of business culture and risk tolerance, 7)
  
  
  Example 2:
  Text:
  to ask Dr. Jemison that I think for her, and for me, space was this idea that really inspired us, I think, to go explore new frontiers. You know, it was this imagination of this thing that, you know, for me at least, like made me want to study computer science, because – like that was the most interesting terrestrial frontier I could go explore. And like you know, the thing that I wonder about is like what that frontier is, like what that inspiration will be for the next generation of scientists, and engineers and explorers. You know, like maybe it’s synthetic biology, but it’s going to be interesting to see whatever it is. [MUSIC] CHRISTINA WARREN: I couldn’t agree more. I look forward to watching and learning from all of that. All right, well, that’s a wrap. Thank you so much to Mae for joining us today. And to our listeners. Thank you for joining us and
  
  Output:
  (“entity”, SPACE, CONCEPT, Space is described as an inspiring concept that motivates exploration and study in new frontiers, particularly in science and technology)
  (“entity”, COMPUTER SCIENCE, FIELD, Computer science is highlighted as an interesting terrestrial frontier that the speaker was motivated to explore due to the inspiration from space)
  (“entity”, SYNTHETIC BIOLOGY, FIELD, Synthetic biology is mentioned as a potential inspiring frontier for the next generation of scientists, engineers, and explorers)
  (“entity”, CHRISTINA WARREN, PERSON, Christina Warren is the speaker who expresses agreement and looks forward to learning from the developments in new scientific frontiers)
  (“entity”, MAE, PERSON, Mae is mentioned as a guest who joined Christina Warren in the discussion about future scientific frontiers)
  
  (“relationship”, SPACE – COMPUTER SCIENCE, Space as a concept inspired the speaker to study computer science, 8)
  (“relationship”, CHRISTINA WARREN – MAE, Christina Warren thanks Mae for joining the discussion, 7)
  
  
  Example 3:
  Text:
  educational outcomes for kids. And if you look at the children of immigrants in East San Jose or East Palo Alto here in the Silicon Valley, like often, the parents are working two, three jobs. Like, they’re so busy that they have a hard time being engaged with their kids. And sometimes they don’t speak English. And so, like, they don’t even have the linguistic ability. And you can just imagine what a technology like this could do, where it really doesn’t care what language you speak. It can bridge that gap between the parents and the teacher, and it can be there to help the parent understand where the roadblocks are for the child and to even potentially get very personalized to the child’s needs and sort of help them on the things that they’re struggling with. I think it’s really, really very exciting. BILL GATES: Yeah, just the language barriers, we often forget about that. And that comes up in the developing world. India has
  
  Output:
  (“entity”, EAST SAN JOSE, GEO, A region in Silicon Valley where many immigrant families reside, and parents often work multiple jobs)
  (“entity”, EAST PALO ALTO, GEO, A region in Silicon Valley known for its significant immigrant population and economic challenges)
  (“entity”, SILICON VALLEY, GEO, A major hub for technology and innovation in California, USA)
  (“entity”, BILL GATES, PERSON, Prominent technology leader and philanthropist who discusses the impact of technology on overcoming language barriers)
  (“entity”, TECHNOLOGY, CONCEPT, Refers to new technological solutions that can assist in bridging language gaps between parents and teachers, and provide personalized support to children)
  
  (“relationship”, EAST SAN JOSE – SILICON VALLEY, East San Jose is a part of Silicon Valley, 9)
  (“relationship”, EAST PALO ALTO – SILICON VALLEY, East Palo Alto is located within Silicon Valley, 9)
  (“relationship”, TECHNOLOGY – BILL GATES, Bill Gates discusses the potential of technology to solve language barriers and educational challenges, 8)
  
  
  Real Data
  ######################
  
  Text: {input_text}
  
  Output:

Here, the automatically generated prompt using the sample content from Code Sample 2 identifies fourteen entity examples across six different entity types (person, location, group, concept, field, and geography) and eight relationship examples.

To assess how this impacts the extraction of the entire dataset, we used both the default and the auto-tuned prompt to generate the entity and relationship outputs. Before we explain the results, let’s review the default prompt’s outputs, which produced seven entities and six relationships, as shown in Code Sample 4. 

Code Sample 4: Default extraction output

Entities and Relationships:
(“entity”, ASHLEY LLORENS, PERSON, Ashley Llorens is a scientist, engineer, hip-hop artist, and vice president at Microsoft. He has worked at Johns Hopkins Applied Physics Laboratory and was nominated by the White House to serve as an AI expert for the Global Partnership on AI. He is also a voting member of the Recording Academy for the Grammy Awards.)
(“entity”, JOHNS HOPKINS APPLIED PHYSICS LABORATORY, ORGANIZATION, An institution where Ashley Llorens worked for two decades developing novel AI technologies and served as the founding chief of the intelligent systems center.)
(“entity”, WHITE HOUSE OFFICE OF SCIENCE AND TECHNOLOGY POLICY, ORGANIZATION, A U.S. government office that nominated Ashley Llorens to serve as an AI expert for the Global Partnership on AI.)
(“entity”, GLOBAL PARTNERSHIP ON AI, ORGANIZATION, An international initiative focused on AI where Ashley Llorens serves as an expert.)
(“entity”, RECORDING ACADEMY, ORGANIZATION, An organization responsible for the Grammy Awards, of which Ashley Llorens is a voting member.)
(“entity”, MICROSOFT, ORGANIZATION, A multinational technology company where Ashley Llorens joined as a vice president, distinguished scientist, and managing director for Microsoft Research.)
(“entity”, KEVIN SCOTT, PERSON, The host of the podcast where Ashley Llorens was interviewed and welcomed to Microsoft.)

(“relationship”, ASHLEY LLORENS – JOHNS HOPKINS APPLIED PHYSICS LABORATORY, Ashley Llorens worked at Johns Hopkins Applied Physics Laboratory for two decades, 9)
(“relationship”, ASHLEY LLORENS – WHITE HOUSE OFFICE OF SCIENCE AND TECHNOLOGY POLICY, Ashley Llorens was nominated by the White House Office of Science and Technology Policy to serve as an AI expert, 8)
(“relationship”, ASHLEY LLORENS – GLOBAL PARTNERSHIP ON AI, Ashley Llorens serves as an AI expert for the Global Partnership on AI, 8)
(“relationship”, ASHLEY LLORENS – RECORDING ACADEMY, Ashley Llorens is a voting member of the Recording Academy for the Grammy Awards, 7)
(“relationship”, ASHLEY LLORENS – MICROSOFT, Ashley Llorens joined Microsoft as a vice president and managing director for Microsoft Research, 9)
(“relationship”, KEVIN SCOTT – ASHLEY LLORENS, Kevin Scott interviewed Ashley Llorens on the podcast and welcomed him to Microsoft, 8)

Using the auto-tuned, domain-specific, automatically generated prompt, we achieved a deeper extraction, producing nine entities and eight relationships, as shown below in Code Sample 5. 

Code Sample 5: Auto-tuned extraction output

Entities and Relationships:
(“entity”, ASHLEY LLORENS, PERSON, Ashley Llorens is a scientist, engineer, hip-hop artist, and executive at Microsoft. He has a background in AI technology development and has been involved in the music industry as a hip-hop artist and Grammy voting member)
(“entity”, KEVIN SCOTT, PERSON, Kevin Scott is the host of the podcast where Ashley Llorens is a guest. He engages with Ashley about his background and career)
(“entity”, JOHNS HOPKINS APPLIED PHYSICS LABORATORY, ORGANIZATION, An institution where Ashley Llorens worked for two decades, developing novel AI technologies and leading the intelligent systems center)
(“entity”, WHITE HOUSE OFFICE OF SCIENCE AND TECHNOLOGY POLICY, ORGANIZATION, A governmental office that nominated Ashley Llorens to serve as an AI expert for the Global Partnership on AI)
(“entity”, GLOBAL PARTNERSHIP ON AI, ORGANIZATION, An international initiative focused on AI where Ashley Llorens serves as an expert.)
(“entity”, RECORDING ACADEMY, ORGANIZATION, An organization responsible for the Grammy Awards, of which Ashley Llorens is a voting member.)
(“entity”, MICROSOFT, ORGANIZATION, A major technology company where Ashley Llorens recently joined as a vice president, distinguished scientist, and managing director for Microsoft Research)
(“entity”, CHICAGO, LOCATION, The city where Ashley Llorens grew up, specifically mentioned as the south side and south suburbs, which influenced his interest in music and technology)
(“entity”, HIP-HOP, MUSIC GENRE, A music genre that significantly influenced Ashley Llorens during his childhood in Chicago, leading him to pursue a career in music alongside his technical career)

(“relationship”, ASHLEY LLORENS – JOHNS HOPKINS APPLIED PHYSICS LABORATORY, Ashley Llorens worked at Johns Hopkins Applied Physics Laboratory for two decades, developing AI technologies, 9)
(“relationship”, ASHLEY LLORENS – WHITE HOUSE OFFICE OF SCIENCE AND TECHNOLOGY POLICY, Ashley Llorens was nominated by the White House Office of Science and Technology Policy to serve as an AI expert, 9)
(“relationship”, ASHLEY LLORENS – GLOBAL PARTNERSHIP ON AI, Ashley Llorens serves as an AI expert for the Global Partnership on AI, 9)
(“relationship”, ASHLEY LLORENS – RECORDING ACADEMY, Ashley Llorens is a voting member of the Recording Academy for the Grammy Awards, 7)
(“relationship”, ASHLEY LLORENS – MICROSOFT, Ashley Llorens recently joined Microsoft as a vice president and managing director for Microsoft Research, 9)
(“relationship”, ASHLEY LLORENS – CHICAGO, Ashley Llorens grew up in Chicago, which influenced his early interest in music, particularly hip-hop, 7)
(“relationship”, ASHLEY LLORENS – HIP-HOP, Ashley Llorens was deeply influenced by hip-hop music during his upbringing in Chicago, leading him to pursue a career in music, 8)
(“relationship”, KEVIN SCOTT – ASHLEY LLORENS, Kevin Scott hosts Ashley Llorens on the podcast, discussing his background and career transitions, 7)

Compared with the default prompt, the auto-tuned prompt is an improvement, with more entities and more relationships, providing a more comprehensive view of our data. One key difference between this output and the output from the default prompt is the expansion in entity types being extracted. The default prompt is limited to three example types: organization, geography, and person. However, the auto-tuned prompt expands to more example types derived from the sample input text: organization, person, location, and music genre. 

Putting it all together 

We can observe a clear difference in the final outputs after using these auto-tuned prompts for indexing the podcast source data. To measure this difference, we compared the size of the resulting knowledge graphs using default with auto-tuned prompts. The following results were achieved while keeping all parameters constant between both runs and using GPT4-Turbo: 

 Entities Relationships Communities Default prompt 1796 2851 352 Auto-tuned prompt 4896 8210 1027 

As shown, the use of auto-tuning yields a significantly larger knowledge graph. For example, a prompt that looks for molecules will extract much more from a chemistry dataset than one that looks for people and places. More communities in the knowledge graph means that it can better serve global search queries. While the size of the knowledge graph can be a proxy for its utility, the true measure of utility is observed in end-to-end queries. To demonstrate this, we provide the following side-by-side comparison of a question using the default and auto-tuned prompts.

We asked GraphRAG the following question: “What’s the relationship between Ashley Llorens and Chicago?” 

First, we got the following results using the knowledge graph built with the default prompt. We then used the knowledge graph built with our auto-tuned prompt:

Default query results 

“I am sorry but I am unable to answer this question given the provided data.“

Auto-tuned query results 

“**Origin and Influence**: Ashley Llorens has a significant connection to Chicago, which has deeply influenced his personal and professional life. Growing up in a city known for its vibrant music scene, particularly hip-hop, has shaped his musical pursuits and artistic expression.“

“**Professional Impact**: His background and experiences in Chicago have carried over into his professional environment, notably at the Applied Physics Laboratory, where he has integrated elements of hip-hop culture [Data: Reports (940)].“

With auto-tuned indexing prompts, our knowledge graph became more representative of the dataset’s entities and relationships, enabling it to yield a valid response to the query. 

Advancing auto-tuning and expanding GraphRag’s capabilities

Looking ahead, we’re exploring other methods to build on this auto-tuning work. We’re excited to investigate new approaches for creating the core GraphRAG knowledge graph and are also studying ways to measure and evaluate the quality of these graph structures. Additionally, we’re researching methods to better assess performance so that we can identify the types of queries where GraphRAG provides unique value. This includes evaluating human-generated versus auto-tuned prompts, as well as exploring potential improvements to the auto-tuner. 

Overall, these new auto-tuner developments make GraphRAG much more accessible and turnkey. We hope this auto-tuning work removes many of the challenges involved when working with new datasets. We invite you to try out these capabilities yourself using GraphRAG’s core library (opens in new tab) and our Azure-based solution accelerator, available on GitHub (opens in new tab).

Try out GraphRAG Try out GraphRAG Accelerator Opens in a new tab

The post GraphRAG auto-tuning provides rapid adaptation to new domains appeared first on Microsoft Research.

Situs judi IDN Slot online yang resmi dan terbaik adalah tempat untuk player yang ingin melakukan taruhan dengan cara online. di dalamnya kamu akan menemukan permainan sabung ayam yang sudah terkenal di Indonesia. game sabung ayam sendiri adalah permainan yang sangat disukai oleh para pecinta ayam aduan tidak hanya di Indonesia saja tapi juga di berbagai belahan dunia. Dikarenakan adanya larangan perjudian, sekarang seluruh pecinta ayam aduan melakuka taruhan dengan sistem online. karena itu kamu bisa mencoba game ini di agen judi resmi dan terpercaya untuk dapatkan keseruan tanpa batas di dalamnya.

Beragam Keunggulan Yang Ditawarkan Situs Sabung Ayam Online Resmi Indonesia

Kebanyakan petaruh di Indonesia yang melakukan taruhan sabung ayam diwajibkan untuk memilih agen atau situs judi sabung ayam terbaik terlebih dahulu. Karena ketika pemilihan agen dapat dilakukan oleh player, tentu saja hal ini akan memudahkan jalannya dalam mendapatkan keuntungan dengan mudah. agen judi sabung ayam terbaik sendiri menawarkan beberapa keungguln yang membuat petaruh suka dan jatuh hati saat bermain di dalamnya. berikut ini sudah ada keunggulan yang akan kamu temukan di dalam situs judi sabung ayam resmi untuk para player di indonesia:

• Fitur Live Streaming

Untuk keunggulan yang akan kamu dapatkan pertama kali adalah live streaming. Jadi perlu diketahui, lewat fitur yang satu ini, kamu akan menemukan sebuah perlombaan secara langsung. Fitur live streaming memungkinkan para player untuk merasakan sensasi bermain yang sangat mirip seperti pada bandar darat langsung. Karena itu kebanyakan player akan lebih memilih bermain game sabung ayam bersama agen judi yang menyediakan ftur live streaming di dalamnya supaya taruhan lebih menyenangkan.

Para player yang ingin bermain dapat masuk ke dalam pertandingan lewat stus atau aplikasi. Jadi cobalah untuk temukan agen-agen yang memiliki fitur ini di dalamnya. karena ketika kamu ada di dalam sebuah agen judi sabung ayam dengan fitur seperti ini, itu artinya kamu sudah berhasil dapatkan agen terbaik. disini kamu bisa melakukan taruhan dengan aman dan nyaman serta mendapatkan hasil yang begitu menggiurkan.

• Hadir untuk semua kalangan

Kemudian, kamu juga akan menemukan banyak sekali player yang ikut bermain di dalam agen judi seperti ini. karena itu, game ini hadir untuk semua kalangan player yang membuatnya semakin populer. Game ini bisa diakses dengan mudah oleh player karena alat main yang digunakan hanyalah sebuah smartphone yang dihubungkan ke jaringan internet saja.

Jadi apabila kamu sudah menemukan jaringan internet di dalam smartphone milik kamu, kamu bisa akses sabung ayam online kapan saja dan dimana saja. kamu juga dapat menikmati permainan ini dengan penawaran tanpa batas yang membuat game ini sangat sayang bila dilewatkan begitu saja. jadi cobalah untuk melakukan pemilihan situs sabung ayam sampai menemukan agen seperti ini.

• Terjamin kEamanannya

Dan yang terakhir adalah mendapatkan game sabung ayam yang sudah terjamin keamanannya. Ini adalah salah satu keunggulan yang juga akan kamu dapatkan dari situs judi sabung ayam. Jadi apabila saat ini kamu mengikuti taruhan sabung ayam secara online, keamanan yang ada di dalam agen patut untuk kamu perhatikan dengan benar.

Pasalnya ketika kamu berada di dalam sebuah agen yang keamanannya tidak begitu terjamin, tentu saja kamu harus memperhatikan sistemnya dulu di dalam agen. Karena semua player yang bermain berhak mendapatkan keamanan pada saat berada di dalam agen. Keamanan dan kenyamanan adalah dua hal penting yang akan membantu player untuk bisa dapatkan keuntungan di setiap harinya. player yang bermain game taruhan online juga tidak perlu khawatir jika nanti tidak bisa mendapatkan keseruan pada game yang dimainkan.

Itulah beberapa keunggulan yang akan kamu dapatkan saat berada di dalam agen judi sabung ayam online resmi dan terpercaya. jadi apabila saat ini kamu tertarik dengan game ini, kamu harus temukan situs-situs dengan semua daftar keunggulan di atas untuk dapatkan keuntungan di setiap harinya.

Originally posted 2022-07-12 00:42:47. …

Bermain game judi Joker123 apk online adalah salah satu aktivitas yang saat ini sedang banyak dilakukan oleh player. aktivitas ini disukai oleh player karena bisa mendatangkan penghasilan dalam jumlah yang besar. karena itu, apabila saat ini kamu suka dengan taruhan sabung ayam, pastikan kamu bertaruh dengan strategi. Jika kamu punya strategi untuk bermain game sabung ayam, kesempatan kamu dalam mendapatkan kemenangan akan jauh lebih besar. kamu juga bisa menikmati hasil yang menggiurkan lewat kemenangan yang sudah berhasil diraih.

Berikut Ini Beberapa Strategi Main Sabung Ayam Online Yang Jarang Diketahui Oleh Player

Banyak petaruh mendambakan kemenangan dalam game sabung ayam yang dimainkan. Karena itu, jika kamu salah satunya, maka strategi dalam permainan harus kamu ketahui sejak awal. Jika kamu tahu strategi apa saja yang mesti dilakukan pada saat betting, hal ini akan membantu kamu dalam mendapatkan penghasilan yang besar. nah berikut ini sudah kami rangkum beberapa strategi untuk yang ingin bermain game sabung ayam dengan sistem online:

• Memilih Pertandingan yang Tepat

Dikarenakan ada banyak pertandingan sabung ayam yang akan ditemukan di agen judi terpercaya, maka kamu perlu mencari pertandingan yang memang sudah diketahui dengan baik. Banyanya pertandingan sabung ayam membantu para petaruh untuk memilih yang benar-benar memguntungkan. Jangan pernah berpikir jika semua pertandingan bisa kamu nikmati. jadi sebaiknya cari informasi yang banyak dan lengkap terkait pertandingan yang akan diikuti nanti. Jika sudah mengetahui pertandinganya, barulah kamu bisa dapatkan kemenangan dalam game dengan mudah.

• Amati Hasil Riwayat Pertandingan Terdahulu

Kemudian, strategi kedua untuk player yang ingin bermain game judi sabung ayam adalah mengamati hasil riwayat pertandingan dari kedua ayam yang diadu. Nantinya, kamu akan bertemu dengan ayam berwarna merah dan biru. Disini kamu harus pandai dalam memilih ayam yang dirasa bisa memenangkan pertarungan. Tapi untuk melakukan analisa, dibutuhkan informasi yang lengkap. Kamu bisa perhatikan hasil riwayat dari kedua ayam yang akan diadu.

Biasanya di agen judi sabung ayam online, player bisa menemukan hasil riwayat tersebut dengan mudah. informasi seperti ini tentu saja dibutuhkan oleh player. apalagi yang baru saja masuk ke dalam dunia taruhan adu ayam online itu sendiri. jadi bagi para pecinta ayam aduan, lakukan strategi yang kedua ini dan kamu bisa dapatkan kemenangan dengan mudah.

• Modal Harus Dikelola dengan Baik

Strategi main game sabung ayam yang ketiga adalah modal harus dikelola dengan baik. Jadi buat yang ingin bermain taruhan sabung ayam, kamu harus pastikan jika modal yang akan dikeluarkan sudah melalui perhitungan yang matang. Jangan pernah berpikir jika uang yang kamu punya saat ini bisa kamu jadikan chip. Kamu harus perhatikan dulu berapa jumlah chip yang dibuthkan supaya nanti memudahkan proses deposit yang akan kamu lakukan.

Kebanyakan petaruh pemula langsung bertaruh dengan modal yang banyak.padahal jika hal ini dilakukan akan membuat taruhan yang dilakukan player justru tidak bisa memberikan keuntungan ataupun penghasilan. Maka dari itu, kamu tetap harus membatasi penggunaan modal yang akan dikeluarkan di setiap harinya. karena ini adalah bagian dari strategi yang perlu dilakukan oleh player yang bertaruh. Jika sudah mengaturnya, kerugian besar pasti tidak akan pernah kamu rasakan.

• Bermain dI Situs Terbaik

Dan yang terakhir adalah bermain game sabung ayam di situs judi terbaik. ini merupakan strtegi bermain game judi sabung ayam ketiga yang mesti dilakukan player. jadi untuk yang ingin bermain game sabung ayam, coba pilih dan pilah situsnya dulu. Jika kamu sudah menemukan situs judi terbaik, kamu pasti akan mendapatkan tempat yang bisa berikan kenyamanan untuk playernya.

Itulah beberapa strategi main game judi sabung ayam online yang jarang diketahui oleh player. jadi untuk petaruh yang ingin bermain harus mengikuti strategi di atas untuk bisa dapatkan peluang menang yang besar. jika kamu bisa dapatkan kemenangan dalam permainan sabung ayam, silahkan tarik dananya untuk dapatkan untung menjanjikan. Selamat mencoba dan semoga bermanfaat.

Originally posted 2022-06-08 00:34:09. …

Memang sekarang ini banyak sekali game judi joker123 online yang beredar di internet atau dunia maya dengan begitu bebasnya. Meski game judi dimainkan via onine, tetap saja harus ada modal untuk bisa mengakses dan menikmati keseruan pada game tersebut. begitu pun dengan game judi poker online, semua yang bermain game poker pastinya harus mempelajari dan memahami bagaimana caranya agar modal yang dibawa bisa memberikan hasil yang luar biasa. Karena itu, coba simak beberapa cara di bawah ini untuk pemula yang ingin bermain game poker tapi membawa modal dalamjumlah sedikit.

Hal-Hal Yang Perlu Dilakukan Saat Main Poker Online Memakai Modal Kecil

Permainan poker tidak dapat dipungkiri adalah game judi online yang membutuhkan modal bermain di dalamnya. Modal yang diperlukan pada saat bermain game poker adalah uang asli. Karena itu, ketika kamu berhasil dapatkan kemenangan, maka kemenangan tersebut akan membantu kamu untuk dapatkan penghasilan dalam jumlah yang sangat besar. jadi sudah tidak perlu heran lagi mengapa saat ini banyak petaruh yang bermain game poker dengan modal kecil. Jika kamubisa melakukan taruhan dengan modal kecil, kamu pasti akan bertaruh dengan aman. berikut ada beberapa hal yang sebaiknya dilakukan saat main poker dengan modal kecil:

• MEnguasai Permainannya Dulu

Hal pertama yang mesti dilakukan oleh player pada saat bermain game poker memakai modal kecil adalah menguasai permainannya terlebih dahulu. Jadi disini kamu harus tahu jika penguasaan terhadap permainan judi poker sangat diperlukan oleh player. karena ketika kamu menguasai permainannya dengan baik, akan ada banyak hal positif yang bisa kamu dapatkan nanti.

Jika kamu termasuk salah seorang pemain baru atau pemula, mungkin kamu perlu waktu yang cukup banyak agar bisa mempelajari dan memahami aturan dalam game poker dengan baik. Jika kamu sudah melakukannya, barulah kamu boleh melakukan taruhan dengan uang asli dengan pemahaman yang kamu miliki. Karena kamu pasti bisa mengolah kartu yang didapatkan dengan benar jika penguasaan terhadap permainan sudah kamu dapatkan.

• Memakai Konsentrasi Tingkat Tinggi

Kemudian, kamu juga perlu memakai konsentrasi tingkat tinggi pada saat bermain game poker. Ini adalah hal kedua yang perlu dilakukan oleh player. jangan pernah berpikir jika segala kondsii bisa kamu pakai untuk bermain game judi poker online. pasalnya kamu hanya bisa memenangkan permainan poker jika berada dalam konsentrasi. Kamu harus berkonsentrasi penuh pada taruhan dan fokus dengan segala tahapan yang kamu lalui untuk dapatkan kemenangan dengan mudah.

Kebanyakan player di Indonesia yang bermain tanpa konsentrasi justru akan mengalami kerugian dalam jumlah yang sangat besar. karena itu, kamu harus pastikan jika waktu dan tempat yang dipergunakan untuk bermain sudah tepat. pasalnya hanya dengan cara itu sja, kamu pasti bisa dapatkan taruhan yang lebih gampang untuk dimenangkan.

• Memanfaatkan Trik Jitu

Trik dibutuhkan oleh player pada saat bermain game judi poker. Salah satu trik yang tidakboleh sampai kamu lewatkan adalah trik bluffing atau menggertak. Karena disini kamu harus tahu jika trik bluffing akan sangat membantu kamu untuk mengalahkan player laiin yang duduk di meja taruhan online. jadi trik ini harus kamu lakukan dengan penuh keberanian agar player lain percaya dan segera keluar.

Jika kamu memakai trik yang satu ini, pastikan kamu melakukannya di moment yang tepat. Tidak masalah meski saat ini kartu yang kamu miliki tidak begitu bagus. Jika kamu punya kartu yang tidak terlalu baik nilainya, kamu hanya perlu mengolahnya saja dan berani untuk bluffing. Karena tidak ada satupun player yang bisa mengetahui nilai kombinasi kartu yang kamu dapatkan saat ini. jadi coba untuk melakukan trik yang ketiga ini agar bisa memenangkan permainan denga mudah.

Itulah beberapa hal yang harus dilakukan oleh player bila bermain game judi poker online memakai modal dalamjumlah yang kecil. Jadi apabila saat ini kamu sedang tertarik untuk bermain taruhan poker, kamu boleh melakukan taruhan dengan sejumlah trik di atas. Selamat mencoba dan semoga bermanfaat. 

Originally posted 2022-05-23 00:16:33. …

Beberapa cara sepertinya perlu kamu lakukan apabila ingin bermain game judi idnplay download online dengan aman dan nyaman. karena itu, apabila saat ini kamu mengikuti game sabung ayam, pastikan kamu melakukan taruhan dengan cara yang benar. Game ini sudah bisa diakses dan dinikmati dengan cara online. karena itu terdapat kemudahan pada saat mengakses permainannya. Kemudahan dalam mengakses game taruhan sabung ayam tentu saja dikarenakan akses ke dalam game yang hanya membutuhkan smartphone dan internet saja. jadi kamu bisa bermain dimanapun kamu mau dengan mudah.

Beberapa Cara Mengikuti Taruhan Sabung Ayam Online Dengan Aman

Berbeda halnya dengan game sabung ayam yang dimainkan secara langsung, permainan sabung ayam yang kini diakses via online tentu saja jauh lebih aman. karena kamu bisa akses game ini dimana saja yang kamu mau. Hanya dengan smrtphone dan internet saja, akses ke dalam game sudah bisa dilakukan dimanapun kamu mau. Karena itu, rata-rata player lebih suka bermain game sabung ayam dengan sistem online. jadi apabila kamu tetarik, coba simak cara mengikuti taruhan sabung ayam berikut ini agar prosesnya dapat berjalan dengan aman dan nyaman:

• Mendaftar di Situs Judi Resmi

Pertama, kamu harus melakukan pendaftaran di situs judi yang resmi. Ini menjadi cara pertama yang harus kamu lakukan apabila ingin mengikuti taruhan sabung aym secara online. pendaftaran yang dilakukan di dalam agen judi resmi akan membantu kamu supaya bisa dapatkan akun member dengan segera. Data-data yang diberikan ke dalam agen harus data asli. Jangan pernah berpikir jika kamu bisa pergunakan data orang lain pada saat mendaftar di dalam agen sabung ayam.

Siapkan semua data diri yang akan diperlukan pada saat mendaftar. Karena itu, apabila saat ini kamu tertarik untuk bermain nanti, tidak ada salahnya untuk melakukan persiapan yang matang. Pasalnya jika kamu mempersiapkan semuanya dengan matang, hal ini akan membantu kamu supaya bisa menyelesaikan proses daftar dengan mudah. kamu juga bisa mendapatkan akun member tanpa harus dalam waktu yang lama.

• Melakukan Deposit yang Pertama

Kemudian, kamu harus melakukan yang namanya deposit untuk pertama kalinya. Deposit ke dalam stus judi sabung ayam online adalah langkah kedua yang mesti dilakukan oleh player. jadi untuk yang saat ini melakukan transaksi deposit ke dalam agen judi sabung ayam, maka kamu perlu meminta terlebih dahulu nomor rekening agen  lewat cs yang bertugas. Tenang saja, cs akan membantu kamu supaya bisa mendapatkan nomor rekening terbaru milik situs sehingga tidak ada lagi kesalahan yang dilakukan player saat bertaruh.

Deposit ke dalam agen judi sabung ayam sudah semestinya dilakukan di waktu yang tepat. jadi untuk player yang ingin bermain game sabung ayam, jangan pernah bertransaksi jika kamu sendiri tidak tahu apakah bank dalam keadaan online atau tidak. Jadi saat deposit, kamu harus melakukannya ketika bank dalam keadaan online. Dengan begitu, transaksi akan berjalan dengan lancar dan kamu bisa mendapatkan chip untuk bermain taruhan di setiap harinya.

• Memulai Taruhan dengan Bet Kecil

Dan cara terakhir untuk yang ingin mengikuti taruhan sabung ayam adalah memulai taruhan dengan bet kecil. Jadi untuk yang saat ini ingin bermain game sabung ayam, kamu perlu memasang taruhan dengan bet kecil terlebih dahulu. Jangan buru-buru melakukan pemasngan taruhan dengan bet besar. karena kamu akan mengalami kerugian yang besar jika langsung mengikuti taruhan dengan bet besar.

Bermain game sabung ayam dapat dilakukan dengan bet besar dan juga bet kecil. Jika kamu bermain game sabung ayam dengan bet kecil, kemungkinan untuk kamu bisa mendapatkan kemenangan akan jauh lebih besar. berbeda dengan taruhan yang dilakukan dengan bet besar dimana kebanyakan petaruh akan lebih terfokus hanya pada kemenangan dan sisa uang yang dimiliki saja. sehingga mereka lupa dengan kekalahan dan kerugian yang kerap diberikan game ini untuk playernya.

Itulah beberapa cara mengikuti taruhan sabung ayam online yang aman untuk pemula. Jadi supaya kamu bisa bertaruh nanti, coba ikuti satu per satu semua cara main di atas untuk dapatkan untung yang besar.

Originally posted 2022-05-07 00:40:10. …

Dalam bermain game judi poker88 online, tentu kamu harus mengetahui terlebih dahulu sejumlah tips yang akan membantu kamu agar bisa menjalankan taruhan dengan baik. Tips bermain game judi poker sejatinya diperlukan oleh semua player terutama yang masih pemula. Karena ketika tips bermain game poker sudah diketahui oleh player, tentu hal ini akan membantu mempermudah proses taruhan yang akan dilakukan. maka dari itu, coba disimak dulu beberapa tips bermain game judi poker di bawah ini apabila ingin melakukan taruhan dengan mudah dan nyaman.

Beragam Tips Yang Diperlukan Jika Ingin Bermain Game Di Agen Poker Online

Pada saat bermain game judi poker, semua player tentu saja berharap jika mereka bisa mendapatkan hasil keuntungan dalam jumlah besar. tapi sayangnya, sebagai pemula, banyak hal yang sejatinya perlu kamu ketahui terlebih dahulu. Jika kamu tahu banyak hal tentang game yang dimainkan, tentu saja kemungkinan untuk kamu bisa dapatkan kemenangan akan semakin besar. karena itu, coba simak tips bermain game judi poker berikut ini agar kesempatan meraup untung besar akan semakin terbuka lebar:

• Membaca Info Tentang Aturan Main Poker

Pertama, coba baca informasi tentang aturan main game poker yang benar. Jadi untuk yang saat ini suka dengan game judi poker, kamu harus pastika jika informasi di dalam permainan poker sudah kamu dapatkan sejak awal. Banyak hal yang mesti diketahui oleh player salah satunya adalah kombinasi dalam game poker itu sendiri. jadi disini kamu harus mengetahui informasi tentang kombinasi yang ada di dalam game poker supaya bisa dapatkan susunan terbaik pada saat bermain taruhan.

Aturan main game poker lainnya yang perlu diketahui oleh player adalah stategi main yang akan dibutuhkan atau berguna pada saat bermain. jadi kamu harus tahu jika strategi dalam game poker juga dibutuhkan. Salah satu strategi yang sangat populer di dalam dunia betting adalah strategi bluffing. Jadi kamu bisa melakukan bluffing untuk menggertak player lain agar mau keluar dari taruhan yang dimainkan.

• Modal Tampil

Kemudian. Pada saat bermain game judi poker, kamu juga harus punya yang namanya modal tampil. Player yang ingin bermain game poker sudah sepatutnya melakukan deposit terlebih dahulu. Apabila sudah melakukan deposit, barulah uang yang dibawa ke dalam permainan disetorkan ke dalam rekening agen. Dengan uang tersebut, kamu bisa bermain taruhan poker di setiap harinya. kamu bisa mengikuti permainan poker tanpa harus menunggu waktu-waktu tertentu.

Dalam game poker, chip memang begitu dibutuhkan oleh semua player yang bertaruh. Maka dari itu, apabila saat ini kamu tengah tertarik untuk bermain judi poker online, jangan pernah beranggapan jika game poker ini bisa kamu akses atau mainkan tanpa chip atau modal di dalamnya. tanpa adanya modal, game apapun tidak akan bisa diakses termasuk game judi poker itu sendiri.

• Bermain Sabar

Dan yang ketiga adalah bermain game taruhan poker dengan penuh kesabaran. Ini menjadi tips selanjutnya yang tidak boleh dilupakan oleh player di Indonesia. Karena ketika kamu berharap untuk terjun ke dalam game taruhan poker, tentu saja kesabaran menjadi salah satu hal yang sangat dibuthkan disini. Kamu bisa dapatkan banyak kemenangan dan keuntungan bila lebih bersabar dalam menjalankan taruhan online.

Sudah banyak petaruh di Indonesia yang saat ini melakukan taruhan dengan sikap terburu-buru. Bukan hanya memberikan efek kerugian dalam jumlah yang besar saja, jika kamu buru-buru mengikuti kegiatan betting yang ada khawatirnya nanti kerugian dalam jumlah besar juga akan kamu alami nanti. Kesabaran adalah salah satu teknik bermain yang sangat penting untuk dilakukan oleh para player indonesia.

Itulah beberapa tips yang harus dilakukan oleh player apabila ingin bermain bersama agen judi poker online yang terbaik. jadi apabila saat ini kamu mengikuti semua tips bermain di atas, kemugkinan untuk kamu bisa dapatkan keuntungan akan semakin besar. bahkan kamu juga bisa menikmati kesuksesan lewat game ini di setiap harinya. selamat mencoba.

Originally posted 2022-04-10 00:30:54. …

Pembuatan akun member di dalam agen judi idnplay poker online adalah salah satu informasi yang pastinya akan dibutuhkan oleh semua player pemula di indonesia. karena player pemula yang bermain game judi poker akan membutuhkan trik-trik supaya proses pembuatan akun member dapat berjalan mudah dan nyaman. trik membuat akun judi poker sudah sepatutnya dipelajari oleh pemula. Jadi jika kamu salah satu pemula yang saat ini tertarik dengan game poker, coba simak dulu beberapa trik membuat akun judi di bawah ini yang harus dipelajari oleh pemula.

Beragam Trik Untuk Player Yang Ingin Membuat Akun Judi Poker Online

Semua yang sudah terjun ke dalam dunia taruhan pasti ingin mencoba game judi poker yang kini bisa diakses dengan sistem online. terdapat begitu banyak perbedaan yang dimiliki game poker offline dan online. karena itu, jika kamu belum pernah mencoba game ini dengan sistem online, tentu kamu perlu menyimak dulu uraian kali ini. pasalnya banyak sekali hal penting yang sepatutnya diketahui termasuk salah satunya adalah panduan membuat akun judi poker. Berikut ini diantaranya trik membuat akun judi poker yang perlu dipelajari oleh pemula:

• Main di Situs yang Direkomendasikan Orang-orang

Pertama, kamu harus mainkan game judi poker di situs yang sudah direkomendasikan banyak orang. Ini adalah cara main pertama yang perlu dilakukan oleh player. jika kamu bermain game judi poker, kamu tidak boleh salah dalam memilih situs judi. Situs yang dipilih harus situs yang terpercaya. adapun langkah memilih situs poker adalah melihat lisensi resmi dalam situs judi itu sendiri. jadi kamu harus mencari situs judi poker yang sudah mendapatkan sertifikat sebagai situs resmi terlebih dahulu.

Kemudian, kamu juga perlu melakukan taruhan di dalam situs yang sudah memiliki fasilitas dan layanan terlengkap di dalamnya. jadi untuk yang saat ini ingin bermain game judi poker, kamu harus perhatikan dulu apakah situs yang dipilih adlah situs yang sudah dilengkapi dengan pelayanan yang nyaman atau tidak. Karena situs judi terbaik pasti akan memberikan pelayanan terbaik untuk para player yang bermain.

• Siapkan Dana

Kemudian, kamu perlu menyiapkan dana untuk bisa deposit ke dalam situs judi poker online. Bagi yang ingin bermain game poker, kamu perlu memiliki dana dan rekening atas nama kamu sendri. Jika kamu belum membuat rekening bank, silahkan buat dengan memakai nama kamu sendiri. karena pihak agen tidak akan memproses transaksi yang nama akun banknya berbeda dengan nama pemilik akun judi yang dibuat.

• Mengisi Form Data

Langkah ketiga untuk player yang ingin membuat akun judi poker adalah mengisi form data. Jadi apabila kamu sudah menemukan situs judi dan menyiapkan dana yang cukup, ini adalah langkah ketiga yang perlu dilakukan. silahkan isi data-data yang benar. Adapun data yang sebaiknya diisi dengan data kamu sendiri adalah nama akun atau username, nomor rekening bank yang digunakan, jenis bank yang digunakan dan banyak lagi yang lain.

Jika kamu ingin mendapatkan kemudahan dalam melakukan pengisian data diri, usahakan untuk mempersiapkan data-data yang diperlukan. Persiapan data diri sebelum proses pendaftaran dilakukan adalah salah satu langkah yang mesti dilakukan oleh player. karena itu, kamu bisa isi form data dengan cara atau trik satu ini.

• Memulai Taruhan

Dan langkah terakhir yang perlu dilakukan adalah memulai taruhan. ini adalah salah satu langkah membuat akun judi yang terakhir kali mesti dilakukan oleh player. jika kamu sudah memulai taruhan online, itu artinya kamu bisa melakukan taruhan kapan saja. tapi disini kamu harus periksa dulu apakah taruhan yang kamu lakukan sudah kamu mengerti dengan baik atau tidak. Jika tidak, usahakan untuk mempelajari terlebih dahulu aturan main di dalamnya.

Itulah beberapa trik membuat akun judi poker online yang sudah sepatutnya dipelajari dan dipahami oleh semua pemula di Indonesia. jika kamu sudah mempelajarinya, kamu pasti bisa membuat akun member dengan segera. Bahkan waktu yang dibutuhkan nanti hanya beberapa menit saja jika semua langkah sudah benar atau sesuai.

Originally posted 2022-03-25 00:07:14. …

Apabila kamu saat ini begitu tertarik untuk bermain game judi PKVGames online, tentu kamu harus tahu alasan mengapa game ni begitu digilai player di Indonesia. diantara banyaknya games di indonesia saat ini, para player akan lebih suka dengan game sabung ayam. Tentu bukan tanpa alasn mengapa banyak peminat judi online di indonesia suka dengan game ini. Jadi perlu kamu ketahui, game sabung aym adalah game taruhan online yang sangat menarik dan menyenangkan. Jika kamu bermain game sabung ayam, itu artinya kamu tidak perlu repot pergi keluar rumah untuk bisa akses game ini. cukup smartphone dan internet saja, game ini bisa kamu akses dengan mudah.

Berikut Ini Beberapa Alasan Yang Membuat Game Sabung Ayam Online Digilai Oleh Player

Game sabung ayam yang dimainkan secara online bisa kamu akses dan nikmati dengan memakai smartphone dan internet. Jika dulu game ini bisa diakses dengan cara pergi ke sebuah tempat khusus dulu, berbeda dengan sekarang. Kamu sudah bisa mengikuti permainan dengan hanya menggunakan sebuah smartphone. Bahkan akses ke dalam game pun bisa dilakukan dimana saja yang diinginkan. Apabila kamu gemar memainkan game secara online, berikut ini ada beberapa hal menarik yang akan kamu temukan nanti:

• Lebih Murah

Jika dibandingkan dengan permainan judi sabung ayam via offline, game sabung ayam yang dimainkan secara online akan jauh lebih murah. Inilah yang membuat para petaruh sangat suka dengan game judi yang dimainkan secara online. pasalnya untuk mengakses game ini, para player hanya perlu ponsel pintar, internet dan modal puluhan ribu rupiah untuk bisa mengakses game ini. jadi tidak perlu repot melakukan taruhan dengan modal besar karena modal sedikit sudah bisa membantu kamu untuk dapatkan keuntungan yang menjanjikan.

Meski taruhan sabung ayam diakses dengan memakai sistem online, tetap saja para player harus punya modal untuk bertaruh. Karena itu, dengan modal bermain inilah, kamu pasti akan dapatkan taruhan yang lebih menguntungkan. Bukan Cuma itu saja, kamu juga akan dapatkan taruhan yang sangat menyenangkan karena fasilitas betting yang tersedia di dalam agen sabung ayam itu sendiri.

• Banyak Pilihannya

Kemudian, kamu juga akan menemukan lebih banyak pilihan pada saat game sabung ayam dimainkan secara online. pilihan game sabung ayam yang diakses secara online jauh lebih banyak. tak heran jika para player yang bermain game nanti akan menemukan kemudahan dalam memilih jenis yang akan dimainkan. Di dalam agen sabung ayam terpercaya, para player dapat memilih turnamen mana yang akan diikuti pad saat betting nanti.

Maka dari itu, untuk yang saat ini gemar bermain game sabung ayam, jangan pernah melupakan keunggulan yang kedua ini. kamu bisa memanfaatkan beragam pilihan yang ada di dalam agen sabung ayam online terprcaya untuk dapatkan kemenangan dengan mudah. kamu juga bisa dapatkan kesenangan lewat permainan yang kamu ikuti nanti.

• Banyak Bonusnya

Dan yang ketiga adalah menemukan banyak bonus yang untungnya sangat menggiurkan. Jadi untuk para player yang ingin bermain judi sabung ayam, bonus di dalam permainan bisa kamu nikmati di setiap harinya. tapi disini kamu harus tahu jika bonus di dalam agen sabung ayam sangatlah beragam. Kamu akan menemukan bonus mulai dari bonus untuk pendatang baru, bonus bagi yang melakukan deposit d setiap harinya, bonus rollingan dan banyak lagi.

Karena itu, jika kamu ingin bermain game sabung ayam, kamu tidak boleh lupa jika di dalam agen judi sabung ayam terdapat bonus yang akan meningkatkan penghasilan kamu dalam bertaruh. Jika kamu sudah mendapatkan banyak bonus, keuntungan besaar pasti akan segera didapatkan.

Itulah beberapa hal yang membuat permainan judi sabung ayam online lebih digilai player. karena itu, jika kamu sudah melakukan taruhan sabung ayam di dalam agen judi yang terbaik, taruhan akan berjalan dengan mudah. prosesnya sangat menyenangkan sehingga semua petaruh pasti akan betah dan nyaman meski bertaruh dalam waktu lama. Selamat mencoba.

Originally posted 2022-03-03 00:36:34. …

Pada saat bermain game judi login pragmatic play online, para player diwajibkan untuk mengetahui dulu apa saja trik-trik bermain yang akan membantu mereka supaya bisa dapatkan kemenangan dengan mudah. trik bermain game poker tidak hanya dibtuuhkan oleh yang masih pemula saja tapi yang pro dan handal sekalipun akan memerlukan trik supaya bisa dapatkan kemenangan dalam taruhan poker online. ketika kamu bermain dengan sejumlah trik, tentu saja kemenangan pasti akan jauh lebih gampang untuk didapatkan. Karena itu, trik bermain game poker sejatinya begitu dibutuhkan oleh semua player terutama yang masih pemula. Jika kamu salah satunya, coba simak terus pembahasannya.

Trik-Trik Yang Sangat Dibutuhkan Saat Main Game Poker Online

Banyak petaruh di Indonesia yang saat ini dibuat kebingungan lantaran tidak memahami trik bermain game poker dengan baik. Padahal ketika kamu terjun ke dalam taruhan poker yang dinikmati via online,tentu saja hal ini akan membantunya agar bisa dapatkan kemenangan lebih gampang. Sudah menjadi rahasia umum jika player yang bermain game poker ingin dapatkan kemenangan sekaligus terhindar dari kata kekalahan. Karena itu, coba simak beberapa trik berikut ini agar kamu bisa dapatkan kemenangan pada permainan judi poker:

• Pertimbangkan Banyak Hal Terlebih Dahulu

Di dalam permainan judi poker, para player dituntut untuk membuat pertimbangan-pertimbangan terlebih dahulu. Banyak hal yang sejatinya harus dipertimbangkan oleh player terutama setelah melihat kartu yang ada di tangan dan di atas meja. Player yang ingin bermain game poker bisa mengikuti permainan ini dengan sangat mudah dengan cara menghafal semua kombinasi dalam permainan ini. jika sudah bisa menghafal kombinasi dalam permainan poker, hal ini akan membantu player untuk dapatkan kemenangan lebih mudah.

Banyak hal yang tidak boleh dilupakan pad saat memasang taruhan pada permainan poker. Para player yang bermain game poker harus mempertimbangkan jumlah uang yang dikeluarkan. Player yang sudah berada di meja taruhan pun harus tahu dan pandai dalam membuat keputusan apakah akan lanjut atau tidak.

• Cerdas dalam Melakukan Bluffing

Kedua, cobalah untuk cerdas dalam melakukan bluffing. Ini menjadi trik kedua yang bisa dipelajari oleh pemula dan player pro di Indonesia. bluffing sendiri mejadi salah satu teknik yang dikenal oleh para player poker online. pasalnya dari sekian banyak strategi yang ada, sepetrinya kebanyakan player lebih memilih strategi satu ini untuk dapat mengalahkan lawan dalam bermain taruhan poker. Lawan main game poker biasanya akan keluar dari room setelah mendapatkan gertakan seperti ini.

Hanya saja, pada saat membuat gertakan kepada player lawan, pastikan kamu punya kartu yang memang memiliki potensi menang lebih besar. tapi jika tidak, sebaiknya cari moment terbaik untuk kamu bisa melakukan strategi yang satu ini. karena jika kamu tidak punya kartu dengan nilai yang baik tetapi berani melakukan gertakan seperti ini, khawatirnya nanti taruhan yang kamu lakukan justru hanya memberikan kegagalan.

• Bermain dengan Tenang

Dan yang terakhir adalah mainkan game ini dengan penuh ketenangan. Ini juga menjadi salah satu hal yang begitu dibutuhkan oleh semua player yang tengah bermain game poker. Jadi ketenangan dalam bermain game poker dibutuhkan oleh semua petaruh yang ada di indonesia. jadi apabila kamu melakukan taruhan pkoer dengan sistem online, ketenangan dalam bertaruh adalah salah satu hal yang penting untuk dimiliki semua player.

Jika kamu bisa bersikap tenang dan santai pada taruhan yang dijalankan, sudah pasti hasilnya nanti akan jauh lebih baik. Kamu bisa mendapatkan hasil yang sangat maksimal dari taruhan yang dijalankan. Sebab semua petaruh akan mendapatkan kesempatan terbaik mereka dalam meraih kombinasi terbaik jika bermain dengan penuh ketenangan seperti ini.

Demikian informasi mengenai trik poker online yang dibutuhkan oleh player agar bisa dapatkan banyak kemenangan. Jika kamu bisa mengikuti semua trik dalam permainan poker di atas, tentu hal ini akan membantu kamu untuk bisa dapatkan peluang menang dan untung yang jauh lebih menjanjikan. Selamat mencoba dan semoga bermanfaat.

Originally posted 2022-01-09 00:21:19. …

Permainan judi s128 apk adalah salah satu game yang diminat oleh kalangan penjudi di Indonesia. peminat game ini datang dari semua kalangan. Jadi tak heran jika kebanyakan petaruh di Indonesia lebih suka dengan game ini dibanding permainan judi online yang lain. apabila kamu tertarik untuk bermain game poker dengan cara online, pastikan kamu sudah paham dengan panduan dasar di dalamnya terlebih dahulu. Pemahaman tersebut akan sangat berguna supaya kamu bisa dapatkan keuntungan besar dengan cara yang mudah.

Beragam Trik Agar Dapatkan Kemenangan Dalam Game Judi Poker Online

Cara menentukan pemenang dalam game ini adalah player yang bisa bertahan sampai akhir. Jadi apabila kamu bermain game judi poker, pastikan kamu selalu bertahan hingga di level akhir atau sampai tidak ada satu pun player yang duduk bersama kamu di dalam meja taruhan poker. Maka dari itu, usahakan untuk dapat melakukan taruhan dengan benar sehingga kemenangan bisa segera kamu dapatkan. Nah, bagi yang masih baru atau pemula dan belum pernah mempelajari trik bermain game poker, berikut ini sudah kami rangkum beberapa trik dapatkan kemenangan yang perlu kamu lakukan:

• Membawa modal yang cukup

Untuk trik bermain game judi poker yang pertama adalah membawa modal yang cukup. Jadi untuk para playeryang ingin bermain game poker, pastikan kamu sudah mengetahui berapa jumlah modal yang akan dibawa ke dalam taruhan. pasalnya modal memegang peranan penting untuk player yang ingin bertaruh. Jika kamu punya modal yang cukup, kemenangan bukanlah hal yang sulit didapat. Bahkan dengan modal yang kamu miliki, kamu juga bisa mainkan game poker berulang kali dengan mudah dan praktis.

Nah, untuk player yang saat ini ingin bermain game poker, kamu harus lakukan yang namanya deposit dulu. Jadi uang yang kamu punya bisa disetorkan ke rekening milik situs. Jika sudah disetorkan, modal yang semula berupa uang asli akan berubah menjadi chip. Dengan chip ini, kamu bisa bermain taruhan kapan saja kamu mau. Bahkan kamu juga bisa mengakses dan menikmati permainan dimana saja dengan mudah dan nyaman.

• Berpindah Posisi

Kemudian, trik jitu kedua untuk player yang ingin dapatkan kemenangan dalam game judi poker online adalah berpindah posisi. Ini adalah salah satu cara terbaik untuk player yang ingin menang. Jadi buat player yang ingin bermain game poker, kamu boleh berpindah ke kursi yang lain jika kursi yang kamu duduki saat ini masih belum bisa memberikan kamu banyak kemenangan.

Sudah banyak petaruh yang berhasil dapatkan kemenangan ketika mereka mengambil semua peluang yang ada. Jadi apabila saat ini kamu ingin bermain game poker, berpindah posisi menjadi langkah kedua yang tidak boleh sampai kamu lewatkan begitu saja. kamu harus pindah ke posisi atau meja yang lain untuk dapatkan kesempatan menang lebih besar.

• Tentukan Waktu yang Pas

Dan yang terakhir adalah menentukan waktu yang pas untuk bermain taruhan online. waktu bermain game judi poker sebenarnya bisa kapan saja. mengingat game ini selalu tersedia 24 jam di setiap harinya. jadi kamu bisa menikmati game ini kapan saja baik di pagi hari, siang, bahkan di waktu dini hari sekalipun.

Jika kamu ingin mengejar kemenangan dalam taruhan poker, maka waktu yang pas adalah salah satu solusi yang tepat untuk kamu bisa dapatkan keuntungan. Jadi buat yang ingin bermain game poker, pemilihan waktu disini begitu dibutuhkan oleh player. Jika kamu sudah berhasil menemukan waktu yang tepat, kemungkinan untuk kamu dapatkan kemenangan akan semakin mudah. kamu bisa bermain dengan penuh dan fokus jika waktu yang digunakan saat bettingsudah tepat.

Itulah beberapa trik jitu untuk dapatkan kemenangan dalam game judi poker online. jadi apabila saat ini kamu sudh mengikuti semua trik di atas dengan baik, coba lihat hasil yang didapatkan. Karena petaruh yang bermain dengan trik yang tepat akan mendapatkan hasil keuntungan yang jauh lebih besar. selamat mencoba.

Originally posted 2022-01-08 00:14:24. …

Hari-hari ini, kita menghabiskan lebih banyak waktu di internet daripada sebelumnya. Salah satu konsekuensi yang tidak diinginkan dari revolusi digital adalah semakin populernya hiburan online. Judi Download Joker123 online adalah salah satu hiburan yang menarik perhatian orang-orang di seluruh dunia. Bermain judi secara onlineatau di perangkat seluler telah menjadi cara paling populer untuk bersantai bagi banyak orang. Judi berbasis online  berada di garis depan tren ini, menawarkan hiburan yang cepat, menyenangkan, dan objektif. Pada artikel ini, kami akan menunjukkan kepada kamu bagaimana memilih permainan atau game pada judi dengan baik.

Cara Pilih Permainan Judi Online

Beragam cara dapat kamu lakukan apabila ingin mendapatkan permainan judi yang terbaik. meski tidk mudah, tapi pemilihan game judi adalah salah satu hal yang mesti dilakukan. berikut ini sudah kami rangkum informasi untuk player yang ingin memilih permainan pada judi secara online. Beberapa diantaranya adalah sebagai berikut.

• Pilih Yang Bisa Menghasilkan Banyak Uang

Sebelum kamu dapat menemukan permainan yang sempurna untuk diri kamu sendiri, kamu harus terlebih dahulu mengidentifikasi komponen terpenting dari permainan kamu. Tidak mengherankan jika banyak pemain ingin menghasilkan uang sebanyak mungkin, tetapi bagaimana kamu memilih permainan yang tepat untuk itu? Jika kamu mencari keuntungan yang cukup besar untuk berhenti dari pekerjaan harian kamu, kamu disarankan untuk mencari permainan judi online yang populer karena kerap memberikan hasil besar untuk para playernya.

Biasanya game-game seperti ini adalah game yang hanya mengedepankan hoki semata. Tapi meski bisa memberikan hasil yang besar, tetap saja para player yang bertaruh harus pandai dalam melakukan taruhan dengan teknik yang cemerlang. Ini akan membantu kamu untuk bisa dapatkan kemenangan meski game yang dimainkan adalah game yang hanya mengedepankan keberuntungan.

• Pilih Untuk Mencari Kemenangan Berulang

Beberapa orang memainkan permainan slot termasuk judi dengan harapan memenangkan jackpot besar, sementara yang lain lebih memilih probabilitas tinggi untuk memenangkan jackpot kecil secara teratur. Ada beberapa jenis permainan atau game judui yang dirancang untuk memungkinkan kamu memenangkan hadiah kecil seperti itu secara teratur. Hanya saja, untuk dapat memenangkan game-game yang seperti ini, tetap dibutuhkan teknik dan usaha di dalamnya.

Kamu juga perlu melakukan latihan di setiap harinya supaya semakin mengenal permainan yang akan diikuti nanti. Karena semua petaruh yang ingin mencari kemenangan berulang harus pandai dan jeli dalam memilih permainan. Jika berhasil temukan kemenangan berulang dalam taruhan yang dimainkan, kamu pasti akan meraup hasil besar. meski bermodalkan puluhan ribu, tetap saja untung hingga jutaan rupiah bisa segera kamu nikmati dengan mudah. 

• Maksimalkan permainan yang sudah dipilih

Bagi yang sudah berhasil menemukan permainan judi terbaik. pastinya kamu harus memaksimalkan keuntungan dalam taruhan yang dimainkan. Apalagi game yang kamu akses dan nikmati saat ini adalah game yang memakai uang asli di dalamnya. karena itu, penggunaan uang asli dalam game taruhan online sudah sepatutnya menjadi motivasi untuk kamu melakukan taruhan dengan giat.

Karena hanya dengn cara ini saja, kamu bisa dapatkan kesempatan besar dalam meraih yang namanya kemenangan. Kamu juga bisa meningkatkan penghasilan kamu dalam bertaruh ketika game-game yang dipilih adalah game yang sudah sesuai dengn basic atau kapasitas kamu sebagai seorang petaruh. 

• Berhenti Setelah Menang

Jangan lupa, setelah kemenangan besar berhasil didapatkan, hentikan segera taruhannya. Karena ketika kamu menghentikan taruhan yang kamu mainkan, hal ini akan sangat membantu kamu untuk dapatkan hasil dari kemenangan yang sudah diraih. Maka dari itu, sudah sepatutnya untuk kamu menghentikan semua taruhan yang kamu lakukan apabila kemenangan berhasil dicapai. Tapi disini kamu harus tetapkan target menang agar tahu kapan harus berhenti dari taruhan. 

Itulah tadi beberapa cara menentukan permaiann dalam hal bermain judi online. Semoga artikel ini dapat bermanfaat untuk kamu, khususnya yang ingin memulai keberuntungan pada permainan sejenis ini. Ingatlah untuk terus memastikan jika kamu telah tepat dalam mengambil langkah karena jika salah sedikitpun maka kamu akan menemui kerugian.

Originally posted 2022-09-01 00:46:49. …

Ce qui est toujours plaisant lors de montée de version d’un outil, restera toujours la découverte des nouveautés  et ensuite … le recherche délicate des fonctions d’origine … sans parler des fonctions désormais absentes … A chaque migration, l’outil ...

Il est vraiment difficile d’arriver à suivre toutes les nouveautés et autres informations diverses et variés sur SharePoint et Office 2016, On peut passer à coté de certaines …pas d’une nouveauté mais plutôt d’une disparition, d’une feature deprecate...

The lower layers in the modern computing infrastructure are written in languages threatened by exploitation of memory management errors. Recently deployed exploit mitigations such as control-flow integrity (CFI) can prevent traditional return-oriented programming (ROP) exploits but are much less effective against newer techniques such as Counterfeit Object-Oriented Programming (COOP) that execute a chain of C++ virtual methods. Since these methods are valid control-flow targets, COOP attacks are hard to distinguish from benign computations. Code randomization is likewise ineffective against COOP. Until now, however, COOP attacks have been limited to vulnerable C++ applications which makes it unclear whether COOP is as general and portable a threat as ROP. This paper demonstrates the first COOP-style exploit for Objective-C, the predominant programming language on Apple’s OS X and iOS platforms. We also retrofit the Objective-C runtime with the first practical and efficient defense against our novel attack. Our defense is able to protect complex, real-world software such as iTunes without recompilation. Our performance experiments show that the overhead of our defense is low in practice.

Modeling relation paths has offered significant gains in embedding models for knowledge base (KB) completion. However, enumerating paths between two entities is very expensive, and existing approaches typically resort to approximation with a sampled subset. This problem is particularly acute when text is jointly modeled with KB relations and used to provide direct evidence for facts mentioned in it. In this paper, we propose the first exact dynamic programming algorithm which enables efficient incorporation of all relation paths of bounded length, while modeling both relation types and intermediate nodes in the compositional path representations. We conduct a theoretical analysis of the efficiency gain from the approach. Experiments on two datasets show that it addresses representational limitations in prior approaches and improves accuracy in KB completion.