In the dynamic world of Linux administration, achieving advanced specialization is not just a career booster; it's a strategic imperative. The LPIC-3 High Availability and Storage Clusters (Exam 306-300) is a pinnacle certification from LPI, designed for senior Linux professionals aiming to master the complexities of high availability and robust storage solutions. This guide is crafted to provide a comprehensive, strategic roadmap, helping you understand not just how to achieve a desirable 306-300 passing score, but also to build a profound expertise in critical enterprise-level Linux deployments.
Many aspiring LPIC-3 specialists often focus solely on memorizing facts, hoping to scrape by the 306-300 passing score. However, the LPIC-3 306 exam, officially known as the LPI High Availability and Storage Clusters exam, demands a deeper understanding and practical application of concepts. This specialization decision guide will navigate you through the exam's intricacies, from its core objectives and syllabus topics to effective preparation strategies and the invaluable career benefits it offers. Prepare to transform your approach from merely chasing a score to mastering the essential skills that truly define an expert in Linux Professional High Availability.
Why LPIC-3 High Availability and Storage Clusters?
High availability (HA) and robust storage are the bedrock of modern IT infrastructure. As businesses increasingly rely on continuous service delivery, the demand for Linux professionals who can design, implement, and maintain highly available systems and scalable storage solutions has skyrocketed. The LPIC-3 High Availability and Storage Clusters certification directly addresses this critical need, validating your ability to keep mission-critical services running with minimal downtime.
The Strategic Advantage of LPIC-3 306
Earning the LPIC-3 306 certification signifies a profound level of expertise. It demonstrates your capability to manage complex clustering technologies, implement distributed storage solutions, and ensure business continuity. This isn't just about understanding theory; it's about practical, hands-on knowledge that translates directly into real-world value. Certified professionals are better equipped to reduce operational risks, optimize resource utilization, and architect resilient systems, making them indispensable assets to any organization. The certification enhances your credibility and opens doors to advanced roles in system architecture, DevOps, and site reliability engineering.
Furthermore, in a competitive job market, an LPIC-3 specialization sets you apart. Employers are actively seeking individuals who possess verified skills in these areas. The LPI certification ecosystem, particularly at the LPIC-3 level, is widely recognized globally, ensuring that your credential holds significant weight across various industries and geographies.
Who Should Consider the LPI 306-300 Exam?
The LPI 306-300 exam is designed for experienced Linux administrators, system architects, and IT professionals who already hold an active LPIC-2 certification. It assumes a solid foundation in general Linux administration and networking. If your career trajectory involves:
- Designing and implementing highly available services.
- Managing and deploying clustered file systems.
- Working with advanced storage solutions like GlusterFS and Ceph.
- Minimizing downtime and ensuring business continuity for critical applications.
- Leading infrastructure projects in cloud or on-premises environments.
Then the LPIC-3 High Availability and Storage Clusters certification is an excellent strategic choice. It's a challenging exam that requires dedication and hands-on experience, but the rewards in terms of skill development and career advancement are substantial. This guide aims to prepare you for the journey, equipping you with the knowledge needed to approach the exam with confidence and secure a strong 306-300 passing score.
Understanding the LPI 306-300 Passing Score and Exam Structure
Before diving into the intricate details of the syllabus, it's crucial to understand the fundamental mechanics of the LPI 306-300 exam. Knowing the format, duration, and the elusive 306-300 passing score can significantly influence your preparation strategy. The exam code is 306-300, and it's officially titled LPI High Availability and Storage Clusters.
Navigating the LPI High Availability and Storage Clusters Certification Cost and Logistics
The LPI 306-300 exam is a premium certification, reflecting its advanced nature. The LPI High Availability and Storage Clusters certification cost is $200 (USD) per attempt. This price is standard across most LPIC-3 specialization exams and represents an investment in your professional future. It's important to factor this cost into your overall preparation budget.
The exam is administered globally through Pearson VUE. You can find more details on how to schedule your LPIC-3 exam and locate testing centers on their website. The registration process is straightforward, requiring you to create an LPI ID if you don't already have one, and then proceeding with the scheduling on the Pearson VUE platform. It is highly recommended to schedule your exam well in advance, giving yourself a firm deadline to work towards.
What to Expect on the LPIC-3 306 V3.0 Exam Content
The LPI 306-300 exam duration is 90 minutes. Within this timeframe, you will be presented with 60 questions. The questions are typically a mix of multiple-choice, fill-in-the-blank, and sometimes short-answer types, designed to test both your theoretical knowledge and practical understanding. The exam evaluates your ability to apply concepts in various scenarios, not just recall facts.
To pass the exam, you need to achieve a 306-300 passing score of 500 out of a maximum of 800. This translates to roughly 62.5% of the questions answered correctly, though the scoring algorithm can be more nuanced than a simple percentage due to varying question weights. The current version of the exam is LPIC-3 306 V3.0, meaning the content reflects the latest developments and best practices in high availability and storage clusters within the Linux ecosystem. Understanding the specific LPIC-3 306 V3.0 exam content is paramount for targeted study.
For a detailed breakdown of the exam objectives and a comprehensive LPI 306-300 exam syllabus, you should refer to the official LPI resources. A great starting point for understanding the curriculum can be found by exploring the detailed exam syllabus.
Deep Dive: LPI 306-300 Exam Syllabus - Mastering High Availability and Storage
The core of your preparation lies in thoroughly understanding and practically applying each topic within the LPI 306-300 exam syllabus. This section breaks down each objective, offering insights into what you need to know to achieve a solid 306-300 passing score. Remember, hands-on experience is key to internalizing these concepts.
High Availability Concepts and Theory (6%)
This foundational section covers the essential principles behind high availability. You need to grasp:
- Definition and Importance: What is high availability, why is it critical for enterprise systems, and how does it differ from fault tolerance and disaster recovery?
- Key Metrics: Understanding Recovery Time Objective (RTO) and Recovery Point Objective (RPO) and their implications for system design.
- Redundancy Principles: Exploring concepts like N+1, N+M, and 2N redundancy models, and their application to different system components (power, network, servers).
- Failure Domains: Identifying single points of failure and strategies to mitigate them. This includes understanding the impact of hardware failures, software bugs, human error, and environmental factors.
- Components of an HA Solution: Familiarity with typical HA architecture components such as heartbeat, cluster managers, shared storage, and monitoring tools.
- Quorum and Fencing: Critical concepts to prevent split-brain scenarios and ensure data integrity in a cluster. You should know how quorum is established and maintained, and the various fencing methods (e.g., power fencing, SCSI persistent reservations).
Practical application involves being able to articulate a design for a highly available service, considering its RTO/RPO requirements and potential failure points. This section sets the stage for all subsequent topics by establishing a common language and understanding of HA goals.
Load Balanced Clusters (8%)
Load balancing is a crucial aspect of high availability, distributing incoming network traffic across multiple servers to ensure responsiveness and reliability. This topic covers:
- Load Balancing Algorithms: Understanding common algorithms like round-robin, least connections, weighted round-robin, and IP hash, and knowing when to apply each.
- Load Balancer Technologies: Familiarity with software-based load balancers such as HAProxy, Nginx, and LVS (Linux Virtual Server). You should be able to configure basic load balancing with these tools.
- Session Persistence: Methods to ensure that a client's requests are always directed to the same server in a session, such as sticky sessions or source IP hashing.
- Health Checks: How load balancers monitor the health of backend servers and remove unhealthy ones from the rotation. This includes various probe types (TCP, HTTP, UDP).
- Scalability vs. High Availability: Understanding the interplay between these two concepts in the context of load balancing. While load balancing improves scalability, it also contributes to HA by redirecting traffic from failed nodes.
- DNS-based Load Balancing: Using DNS records to distribute traffic, its advantages, and limitations.
Hands-on practice configuring a simple web server farm with a load balancer like HAProxy or Nginx is invaluable for this section. Focus on defining backend servers, health checks, and different balancing methods.
Failover Clusters (8%)
Failover clusters are designed to automatically switch to a standby system in case of an active system failure, ensuring continuous service. Key areas include:
- Active-Passive vs. Active-Active: Understanding the differences, benefits, and drawbacks of these two main failover cluster configurations.
- Cluster Resource Management: Tools like Pacemaker and Corosync are central here. You need to know how to install, configure, and manage them.
- Resources and Constraints: Defining resources (IP addresses, services, file systems) and resource groups. Understanding how to set up location, order, and co-location constraints to control resource behavior during failover.
- Fencing and STONITH: Deep dive into various fencing agents (e.g., `fence_apc`, `fence_virsh`) and their role in preventing split-brain conditions.
- Monitoring and Alerts: How to monitor cluster status, identify failures, and configure notifications.
- Service Configuration: Configuring various services (e.g., Apache, MySQL, PostgreSQL) to run as cluster resources.
This is a highly practical section. You should be able to set up a basic Pacemaker/Corosync cluster, configure an IP address and a web service resource, and test failover scenarios. Understanding how to troubleshoot cluster issues is also critical.
DRBD (6%)
DRBD (Distributed Replicated Block Device) is a crucial technology for providing shared storage in a failover cluster without requiring a Storage Area Network (SAN). This topic covers:
- DRBD Basics: Understanding how DRBD replicates block devices in real-time between two servers, acting as network RAID-1.
- Primary/Secondary vs. Dual Primary: Different modes of DRBD operation and their use cases. Focus on the typical active-passive setup with a single primary.
- Configuration: How to configure DRBD resources using `/etc/drbd.conf` and `drbdadm`. This includes defining disk, meta-disk, and network parameters.
- Management Commands: Using `drbdadm` to create, manage, and monitor DRBD resources (e.g., `create-md`, `up`, `down`, `primary`, `secondary`, `status`).
- Integration with Cluster Managers: How DRBD resources are integrated into Pacemaker/Corosync clusters, typically managed as resources that are promoted/demoted during failover.
- Split-Brain Resolution: Understanding how DRBD handles network partitions and the methods for recovering from split-brain.
Setting up a DRBD resource, creating a filesystem on it, and integrating it with a Pacemaker cluster is a fundamental hands-on exercise for this section. Simulating network failures and observing DRBD behavior is also important.
Cluster Storage Access (3%)
This section focuses on how different nodes in a cluster access shared storage. While DRBD provides a block-level solution, other methods are also important:
- Shared Disk Systems: Understanding the concept of shared storage accessed by multiple cluster nodes (e.g., Fibre Channel SAN, iSCSI SAN).
- SCSI Persistent Reservations: How this mechanism is used by cluster managers to control access to shared block devices and prevent data corruption in active-passive setups.
- Multi-Path I/O (MPIO): Configuring and managing redundant paths to shared storage devices for increased reliability and performance. Tools like `multipathd`.
- Network File Systems (NFS/SMB): While not strictly cluster storage in the same vein as SANs, NFS and SMB are often used to provide shared access to files for certain types of cluster applications. Understand their role and limitations in HA contexts.
This topic bridges the gap between traditional storage solutions and cluster management. Focus on how shared storage is presented to cluster nodes and how data integrity is maintained.
Clustered File Systems (4%)
Clustered file systems allow multiple nodes to concurrently access and modify the same file system without data corruption. This is essential for active-active cluster configurations. Key areas:
- Concept of Clustered File Systems: Understanding their role and advantages over traditional file systems when multiple nodes need simultaneous read/write access.
- Types of Clustered File Systems: Familiarity with GFS2 (Global File System 2) and OCFS2 (Oracle Cluster File System 2) and their basic features.
- Locking Mechanisms: How clustered file systems use distributed lock managers to prevent data corruption when multiple nodes write to the same file.
- Configuration Basics: General steps involved in setting up GFS2 or OCFS2 on shared storage, including creating the file system and mounting it.
- Challenges: Understanding performance considerations, complexity, and potential issues when implementing clustered file systems.
While the exam might not require in-depth configuration of GFS2 or OCFS2, understanding their purpose, how they work at a high level, and their use cases is crucial. Practical exposure to one of these file systems is highly beneficial.
GlusterFS Storage Clusters (5%)
GlusterFS is a popular open-source, scalable network file system. It's a key technology for building distributed storage solutions.
- GlusterFS Architecture: Understanding bricks, volumes, and how they combine to form a distributed file system.
- Volume Types: Exploring different volume types like Distributed, Replicated, Striped, and their combinations (e.g., Distributed-Replicated). Know the trade-offs in terms of performance, resilience, and capacity.
- Installation and Configuration: How to set up a basic GlusterFS cluster, peer hosts, and create different types of volumes.
- Client Access: Mounting GlusterFS volumes using the native GlusterFS client or NFS/SMB gateways.
- Management Commands: Using `gluster` CLI to manage volumes (create, start, stop, delete, add/remove bricks).
- Geo-Replication: Basic understanding of how GlusterFS supports replication across geographically dispersed sites for disaster recovery.
Building a small GlusterFS cluster with replicated or distributed-replicated volumes and testing data access and integrity across nodes is a strong recommendation for hands-on practice.
Ceph Storage Clusters (8%)
Ceph is a powerful, highly scalable, and fault-tolerant distributed storage system that provides object, block, and file storage interfaces. Given its complexity and prevalence in modern data centers, this topic carries a significant weight on the exam.
- Ceph Architecture: Understanding the core components: Monitors (MON), Object Storage Devices (OSD), Metadata Servers (MDS), and Ceph Clients.
- CRUSH Algorithm: How Ceph intelligently distributes and replicates data across OSDs using CRUSH (Controlled Replication Under Scalable Hashing).
- Storage Interfaces: Familiarity with Ceph Block Device (RBD), Ceph Object Gateway (RADOS Gateway - RGW), and Ceph File System (CephFS).
- Installation and Deployment: General steps for deploying a Ceph cluster (e.g., using `cephadm` or `ceph-deploy` for older versions).
- Pools and Placement Groups (PGs): Understanding how data is organized into pools and how PGs manage data distribution and replication.
- Basic Management: Using `ceph` CLI to check cluster status, manage OSDs, and interact with pools.
- Failure Domains: How Ceph handles OSD or node failures to maintain data availability and integrity.
Setting up a small Ceph cluster (even in a virtualized environment) and experimenting with RBD volumes and object storage is crucial. This will help demystify its architecture and management.
Hardware and Resource High Availability (2%)
Beyond software, high availability also extends to physical hardware components. This section covers:
- Redundant Hardware: Understanding the importance of redundant power supplies, network interface cards (NICs), and disk controllers.
- Hardware Watchdogs: How hardware watchdog timers can automatically reboot a server if the operating system becomes unresponsive. Configuration of `watchdogd`.
- UPS and Generator Power: Basic understanding of uninterruptible power supplies (UPS) and generators for power redundancy.
- Service Processors/Lights-Out Management: Technologies like IPMI (Intelligent Platform Management Interface) or iLO/DRAC for out-of-band management and remote control of servers.
While often handled at the infrastructure level, a Linux administrator needs to understand how these components contribute to overall system reliability.
Advanced RAID (2%)
RAID (Redundant Array of Independent Disks) provides data redundancy and improved performance at the disk level. This section focuses on advanced aspects:
- Software RAID vs. Hardware RAID: Understanding the differences, advantages, and disadvantages of each. Focus on Linux software RAID (`mdadm`).
- RAID Levels: Beyond basic RAID 0, 1, 5, 6, understanding nested RAID levels like RAID 10 and RAID 50, and their specific use cases, performance characteristics, and fault tolerance.
- Hot Spares: Configuring and managing hot spare disks for automatic rebuilds in `mdadm` arrays.
- Monitoring and Management: Using `mdadm` for monitoring array status, handling failures, and replacing disks.
Practice creating different software RAID levels, simulating disk failures, and performing recovery procedures with `mdadm`. This reinforces the understanding of data protection at the block device level.
Advanced LVM (3%)
LVM (Logical Volume Manager) provides flexible storage management in Linux. This section covers advanced features relevant to HA:
- LVM Snapshots: Creating and managing read-write and read-only snapshots for backups or testing without interrupting the active system.
- LVM Mirroring: Implementing software-level mirroring with LVM for local data redundancy, similar to RAID 1.
- LVM Striping: Understanding how to stripe logical volumes across multiple physical volumes for performance.
- Volume Group Activation: Managing the activation and deactivation of volume groups, especially in clustered environments where only one node should access certain VGs at a time.
- Integration with Cluster Managers: How LVM resources (e.g., activating a VG or mounting an LV) can be managed by Pacemaker to ensure controlled access in a cluster.
Hands-on experience with LVM snapshots and mirroring is key. Understand how LVM commands (`vgchange`, `lvcreate`, `lvdisplay`, `lvs`) are used in an advanced context.
Network High Availability (5%)
Network infrastructure is a critical component of any highly available system. This topic addresses network redundancy and resilience:
- Bonding/Teaming (Link Aggregation): Configuring network interface bonding (e.g., active-backup, balance-rr, 802.3ad) for network redundancy and load balancing.
- VLANs: Understanding Virtual Local Area Networks for network segmentation and isolation, and their role in HA designs (e.g., separate heartbeat networks).
- HAProxy for Network Resilience: While primarily a load balancer, HAProxy can also be configured to provide failover for backend services by detecting and routing around network issues.
- Network Monitoring: Tools and techniques for monitoring network health and performance in a cluster.
- IP Address Takeover: How virtual IP addresses are managed and moved between nodes in a cluster using tools like keepalived or as Pacemaker resources.
- Firewall Configuration in HA: Ensuring consistent firewall rules across cluster nodes and managing stateful connections during failover.
Practice configuring network bonding and virtual IP address failover using either keepalived or Pacemaker. This is a common requirement for highly available services.
Crafting Your LPI 306-300 Study Guide and Preparation Strategy
A successful 306-300 passing score isn't achieved by chance. It requires a structured and dedicated approach. This section outlines how to create an effective LPI 306-300 study guide and implement robust preparation strategies.
Essential Resources: Best Books for LPIC-3 High Availability and Storage Clusters
While LPI does not officially endorse specific books, a combination of authoritative texts and online documentation forms the backbone of a strong study plan. Look for books that cover:
- Linux HA with Pacemaker and Corosync: Dedicated books or extensive guides on these cluster managers are indispensable.
- DRBD Documentation: The official DRBD documentation is very thorough and an excellent resource.
- GlusterFS and Ceph Handbooks: Given their weight on the exam, detailed resources on these distributed storage systems are crucial.
- Advanced Linux System Administration: General texts covering advanced networking, LVM, and RAID will supplement your knowledge.
- Official Documentation: Always refer to the official documentation for specific tools (e.g., `man pages`, project websites).
Combine these with the LPIC-3 High Availability and Storage Clusters exam objectives found on the official LPI website to ensure comprehensive coverage. Remember that technologies evolve, so always seek the most current resources available for LPIC-3 306 V3.0.
Leveraging LPIC-3 306 Practice Exam Questions
LPIC-3 306 practice exam questions are invaluable for gauging your understanding and identifying areas that need more attention. They help you:
- Familiarize with Question Format: Understand the types of questions asked and the expected level of detail.
- Time Management: Practice completing questions within the 90-minute time limit.
- Identify Knowledge Gaps: Incorrect answers highlight topics you need to review.
- Build Confidence: Performing well on practice exams can boost your self-assurance for the actual test.
Look for high-quality practice exams that accurately reflect the difficulty and scope of the LPI 306-300 exam. Some platforms offer realistic simulations, which are highly recommended. However, beware of brain dumps; they hinder true learning and can lead to a false sense of security.
Effective LPI 306-300 Exam Preparation Tips
Beyond theoretical study, here's how to prepare for LPI 306-300 exam comprehensively:
- Hands-on Labs: Set up a virtual lab environment (e.g., using VirtualBox, KVM, or cloud instances) with at least three virtual machines. Practice installing, configuring, and troubleshooting every component of the syllabus. This is arguably the most critical step.
- Study Groups: Collaborating with peers can provide different perspectives, clarify doubts, and make learning more interactive.
- Active Learning: Don't just read. Take notes, draw diagrams, explain concepts to others, and try to implement solutions from scratch.
- Review Official Objectives: Continuously refer to the official LPIC-3 High Availability and Storage Clusters exam objectives on the LPI website. Ensure you can explain each objective thoroughly.
- Time Management: Create a realistic study schedule and stick to it. Break down large topics into smaller, manageable chunks.
- Understand 'What If' Scenarios: The exam often tests your ability to troubleshoot. Think about what happens if a node fails, a network link goes down, or a disk crashes.
- Stay Updated: Linux technologies evolve rapidly. While the V3.0 exam version is specific, staying informed about general trends in high availability and storage can provide valuable context.
- Health and Wellness: Don't neglect sleep, nutrition, and breaks. A well-rested mind performs better.
By following these LPI 306-300 exam preparation tips, you will not only increase your chances of achieving the desired 306-300 passing score but also gain practical skills that are immediately applicable in the workplace. Many find insights into common pitfalls in high availability exam preparation to be very helpful.
Beyond the Exam: Benefits of LPIC-3 High Availability and Storage Clusters Certification
While the immediate goal is to achieve the 306-300 passing score, the long-term benefits of the LPIC-3 High Availability and Storage Clusters certification extend far beyond the exam room. This specialization solidifies your position as a top-tier Linux professional.
Career Progression and Industry Recognition
The benefits of LPIC-3 High Availability and Storage Clusters certification are tangible. It validates advanced skills that are in high demand across various industries. This can lead to:
- Higher Earning Potential: Specialized skills in HA and distributed storage often command higher salaries. The U.S. Bureau of Labor Statistics provides a positive outlook for computer and information technology occupations, where advanced Linux skills are highly valued.
- Advanced Job Roles: Positions such as Senior Linux Administrator, Systems Architect, DevOps Engineer, Site Reliability Engineer (SRE), and Cloud Engineer often require or heavily prefer these competencies.
- Increased Job Security: With expertise in mission-critical systems, you become an indispensable asset, making you more resilient to economic fluctuations.
- Global Recognition: LPI certifications are recognized worldwide, providing portability to your skills and opening up international career opportunities. LPI also integrates with platforms like LPI badges on Credly, further enhancing the visibility of your achievement.
The certification is not merely a piece of paper; it's a testament to your capability to design, implement, and maintain the backbone of modern enterprise IT.
Continuous Learning in Linux Professional High Availability
Achieving LPIC-3 306 is also a stepping stone to continuous professional development. The technologies in high availability and storage clusters are constantly evolving. The knowledge gained from preparing for this exam provides a solid foundation to explore newer technologies, such as advanced cloud-native storage solutions, Kubernetes storage integrations, and further innovations in distributed systems. It fosters a mindset of lifelong learning essential for success in the rapidly changing IT landscape.
Frequently Asked Questions
1. What are the requirements for LPIC-3 High Availability and Storage Clusters?
To achieve the LPIC-3 High Availability and Storage Clusters certification, you must first hold an active LPIC-2 certification. There are no other prerequisites, but strong practical experience with Linux system administration, networking, and storage concepts is highly recommended before attempting the LPI 306-300 exam.
2. What is the LPI 306-300 exam about?
The LPI 306-300 exam, also known as LPIC-3 High Availability and Storage Clusters, focuses on advanced concepts and practical skills required to implement and manage highly available services and distributed storage solutions on Linux. It covers topics like load balancing, failover clusters (Pacemaker/Corosync), DRBD, clustered file systems (GFS2/OCFS2), and distributed storage systems (GlusterFS, Ceph).
3. What is a good 306-300 passing score to aim for?
The official 306-300 passing score is 500 out of 800. While 500 is the minimum, aiming for a score higher than this (e.g., 600-700) indicates a more thorough understanding and provides a comfortable buffer. Focusing on mastering the objectives rather than just hitting the minimum score will ultimately serve your career better.
4. How long is the LPI 306-300 exam duration?
The LPI 306-300 exam duration is 90 minutes. You will need to answer 60 questions within this time, making efficient time management during the exam crucial. Practice exams can help you manage your time effectively.
5. What kind of preparation is best for the LPIC-3 306 exam topics?
The best preparation for the LPIC-3 306 exam topics involves a combination of theoretical study from official documentation and reputable books, extensive hands-on practice in a lab environment for each syllabus objective, and regular use of LPIC-3 306 practice exam questions. Actively building and troubleshooting cluster and storage solutions is paramount.
In conclusion, pursuing the LPI 306-300 certification is a significant step in your journey as a Linux professional. It's more than just achieving a 306-300 passing score; it's about gaining mastery over critical high availability and storage cluster technologies that are at the heart of modern IT infrastructure. By strategically approaching your studies, focusing on practical application, and leveraging the detailed insights provided in this guide, you can confidently navigate the complexities of the LPIC-3 306 exam.
Remember, the goal isn't just to pass, but to truly understand and implement robust solutions. Your commitment to mastering the Linux Professional High Availability exam details will pay dividends in your career, positioning you as an expert capable of handling the most demanding system requirements. Begin your journey today and solidify your expertise in LPI High Availability and Storage Clusters. For further insights into high availability exam challenges, you might find this article on LPIC-3 High Availability exam preparation particularly useful.