DOCSEARCHbyAlgolia
Submit Docs

Feazur Blog

website·Indexed 30 days ago

https://blog.feazur.com

A technical blog featuring guides and tutorials on Flink, Hadoop, Redis, Git, and Mockito, including specific configurations for DolphinScheduler, Flink on YARN, and Flink SQL support for Iceberg.

Tokens
5K
Snippets
21
Records
36
Agent score
50%

What's inside Feazur Blog

  1. Implement Greedy vs Lazy matching

    By default, quantifiers are greedy, meaning they match as much text as possible. To switch to lazy (non-greedy) matching, append a ? immediately after the quantifier.

    Example: .* is greedy; .*? is lazy.

  2. Enable Hive support for Flink SQL

    To use Iceberg with a Hive Catalog in Flink 1.16, you must first enable Hive support. This requires setting the HADOOP_CLASSPATH environment variable and managing specific JAR dependencies in the Flink installation directory.
    # 1. Ensure HADOOP_CLASSPATH is set
# 2. Add the following JARs to FLINK_HOME/lib:
#    - flink-sql-connector-hive-3.1.2.jar (or individual components: 
#      connector-hive_2.12-1.16.2.jar, hive-exec-3.1.0.jar, 
#      libfb303-0.9.3.jar, antlr-runtime-3.5.2.jar)
# 3. Move flink-table-planner_2.12-1.16.2.jar from FLINK_HOME/opt to FLINK_HOME/lib
# 4. Move flink-table-planner-loader-1.16.2.jar out of FLINK_HOME/lib

  3. Group characters and extract substrings

    Grouping () allows treating multiple characters as a single unit. This is essential for applying quantifiers to strings (e.g., (good){3} matches 'goodgoodgood') and for extracting specific parts of a match.

    Extraction in Java:

    • Matcher.group(0): Returns the entire matched string.
    • Matcher.group(n): Returns the substring matched by the n-th group.
    • Matcher.groupCount(): Returns the number of capturing groups in the pattern.

  4. Enable Flink SQL Hive support

    To support Hive within Flink SQL, perform the following file system operations in the Flink installation directory:

    1. Download the flink-sql-connector-hive package and place it in the /lib directory.
    2. Swap the locations of the flink-table-planner package (currently in /lib) and the flink-table-planner-loader package (currently in /opt).

  5. Ensure Redis Cluster Lua scripts access keys on the same node

    When using Lua scripts in a Redis Cluster, you must ensure that all keys accessed by the script map to the same hash slot to avoid the error: lua script attempted to access a non local key in a cluster node.

    Two critical requirements must be met:

    1. Hash Tags: Use curly braces {} in key names (e.g., {distinct}:set and {distinct}:queue). Redis only hashes the part of the key inside the braces, ensuring both keys land on the same slot.
    2. Explicit Key Passing: All keys accessed by the script must be explicitly passed as arguments via the KEYS array. Scripts should not use programmatically generated key names inside the Lua logic. You can verify a key's slot using the CLUSTER KEYSLOT key command.

  6. Create and execute a Flink SQL task in DolphinScheduler

    To execute Flink SQL on a YARN session via DolphinScheduler:

    1. Project Parameter: In Project Management, create a project-level parameter named flink-sql-session-application-id and set its value to the Application ID of a running YARN session.
    2. Task Configuration: Create a workflow and add a Flink component with the following settings:
      • Program Type: SQL
      • Deployment Mode: application
      • Flink Version: >= 1.12
    3. Initialization Script: Use the script to target the YARN session and define the Hive catalog.
    set 'execution.target' = 'yarn-session';
set 'yarn.application.id' = '${flink-sql-session-application-id}';
set 'execution.runtime-mode' = 'batch';
set 'sql-client.execution.result-mode' = 'TABLEAU';

CREATE CATALOG hive_catalog WITH (
  'type' = 'hive';
  'hive-conf-dir' = '/path/to/hive-conf'
);

  7. Capture method arguments using ArgumentCaptor

    Use ArgumentCaptor to capture the arguments passed to a mock method during execution. This allows you to perform assertions on the captured values after the interaction has occurred.
    List mockList = Mockito.mock(List.class);
ArgumentCaptor<String> arg = ArgumentCaptor.forClass(String.class);

mockList.add("one");

// Capture the argument during verification
Mockito.verify(mockList).add(arg.capture());

// Retrieve and assert the captured value
assertEquals("one", arg.getValue());

  8. Enable Iceberg support for Flink SQL

    After enabling Hive support, add the Iceberg Flink runtime JAR to the Flink library directory. For Flink 1.16 and Iceberg 1.2.0, use iceberg-flink-runtime-1.16-1.2.0.jar.

    Caveat for Huawei FusionInsight users: You may encounter NoSuchMethodError due to class dependency conflicts between the Iceberg runtime and FusionInsight. This may require recompiling the iceberg-flink-runtime JAR with updated dependencies.

    # Place the following JAR in FLINK_HOME/lib
cp iceberg-flink-runtime-1.16-1.2.0.jar FLINK_HOME/lib/

  9. Verify method interactions with Mockito

    The verify() method is used to ensure that a specific interaction occurred with a mock object. It can be used to check if a method was called, the number of times it was called, the order of calls, or if no interactions occurred at all.
    // Basic verification
verify(mockList).add("one");

// Verify exact number of calls
verify(mockedList, times(1)).size();

// Verify minimum/maximum calls
verify(mockedList, atLeast(1)).clear();
verify(mockedList, atMost(10)).clear();

// Verify no interactions with the mock
verifyNoInteractions(mockedList);

// Verify a specific method was never called
verify(mockedList, never()).clear();

// Verify interaction order
InOrder inOrder = Mockito.inOrder(mockedList);
inOrder.verify(mockedList).size();
inOrder.verify(mockedList).add("a parameter");

// Verify with flexible arguments
verify(mockedList).add(anyString());

  10. Deploy Flink on YARN using Session Mode

    In Session Mode, a YARN cluster called yarn-session is created before submitting Flink jobs. Multiple jobs can be submitted to this single cluster, sharing a common JobManager and TaskManagers. The lifecycle of the yarn-session is independent of the individual jobs.

    Pros:

    • Resources for multiple Flink jobs within the same session are scalable.

    Cons:

    • Lack of isolation: An exception in one task can cause the entire TaskManager to shut down, affecting other healthy jobs.
    • Higher load on the JobManager as it manages multiple jobs.

  11. Inject mocks into target classes using @InjectMocks

    The @InjectMocks annotation tells Mockito to create an instance of the annotated class and automatically inject any fields annotated with @Mock or @Spy into it.
    @Mock
Map<String, String> wordMap;

@InjectMocks
MyDictionary dic = new MyDictionary();

@Test
public void testInjection() {
    Mockito.when(wordMap.get("aWord")).thenReturn("aMeaning");
    assertEquals("aMeaning", dic.getMeaning("aWord"));
}

  12. Initialize Mockito in Java tests

    Mockito requires initialization before use in a test class. This can be achieved using either a JUnit 5 extension or a manual setup method.
    // Option 1: Using JUnit 5 Extension
@ExtendWith(MockitoExtension.class)
class MyTest {}

// Option 2: Manual initialization in @BeforeEach
@BeforeEach
public void init() {
    MockitoAnnotations.openMocks(this);
}