发布于 由admin

如何利用MySQL的性能架构（Performance Schema）构建一个基于SQL执行阶段的实时性能监控与预警系统？

利用MySQL Performance Schema 构建实时性能监控与预警系统

大家好，今天我们来探讨如何利用MySQL Performance Schema构建一个基于SQL执行阶段的实时性能监控与预警系统。Performance Schema是MySQL官方提供的性能分析工具，它提供了丰富的运行时数据，可以帮助我们深入了解MySQL服务器的内部运行状态，从而进行性能调优和故障排查。

一、Performance Schema 简介

Performance Schema是一个独立的存储引擎，用于收集MySQL服务器的运行时信息。它通过instrumentation技术，在MySQL服务器的关键代码路径上埋点，收集诸如SQL语句执行时间、锁等待、IO操作等信息。这些信息存储在Performance Schema数据库中的表中，我们可以通过SQL语句查询这些表来分析MySQL服务器的性能。

Performance Schema 的优势：

  • 细粒度监控： 能够监控到SQL语句执行的各个阶段，例如parse、optimize、execute等。
  • 实时性： 数据是实时更新的，可以实时监控MySQL服务器的状态。
  • 低开销： Performance Schema的设计目标是低开销，对MySQL服务器的性能影响较小。
  • 可配置性： 可以根据需要启用或禁用特定的instrumentation，以控制监控的粒度和开销。

Performance Schema 的主要组成部分：

  • Instruments： 用于收集特定类型的性能数据。例如，wait/lock/table 用于收集表锁等待的信息。
  • Consumers： 用于消费instrumentation收集的数据，并将其存储到Performance Schema的表中。例如，events_waits_summary_global_by_event_name 表记录了全局的锁等待事件的统计信息。
  • Setup Objects： 用于控制哪些instrumentation是被启用的。例如，setup_instruments 表用于配置哪些instrumentation是启用的。
  • Summary Tables： 用于汇总instrumentation收集的数据，方便进行分析。例如，events_statements_summary_by_digest 表记录了根据SQL语句的摘要信息进行分组的统计信息。

二、系统架构设计

我们的实时性能监控与预警系统将包含以下几个核心模块：

  1. 数据采集模块： 负责从Performance Schema中采集数据。
  2. 数据存储模块： 负责存储采集到的数据。可以选择MySQL自身或其他时序数据库（例如InfluxDB）。
  3. 数据分析模块： 负责分析存储的数据，计算性能指标。
  4. 预警模块： 负责根据预设的阈值，触发预警。
  5. 展示模块： 负责展示监控数据和预警信息。

架构图:

+----------------------+     +----------------------+     +----------------------+     +----------------------+     +----------------------+
| MySQL Server         | --> | Data Collection      | --> | Data Storage         | --> | Data Analysis        | --> | Alerting/Visualization|
| (Performance Schema) |     | (SQL Queries)        |     | (MySQL/InfluxDB)      |     | (SQL/Python/etc.)    |     | (Grafana/Custom UI)   |
+----------------------+     +----------------------+     +----------------------+     +----------------------+     +----------------------+

三、数据采集模块

数据采集模块的核心是编写SQL语句从Performance Schema中查询数据。我们需要根据监控的需求，选择合适的Performance Schema表和列。

示例：采集SQL语句执行时间

以下SQL语句可以从events_statements_summary_by_digest表中获取SQL语句的执行时间统计信息：

SELECT
    DIGEST_TEXT,
    COUNT_STAR,
    SUM_TIMER_WAIT,
    AVG_TIMER_WAIT,
    FIRST_SEEN,
    LAST_SEEN
FROM
    performance_schema.events_statements_summary_by_digest
WHERE
    COUNT_STAR > 0
ORDER BY
    SUM_TIMER_WAIT DESC
LIMIT 10;
  • DIGEST_TEXT: SQL语句的摘要，用于标识SQL语句。
  • COUNT_STAR: SQL语句的执行次数。
  • SUM_TIMER_WAIT: SQL语句的总执行时间（单位是皮秒）。
  • AVG_TIMER_WAIT: SQL语句的平均执行时间（单位是皮秒）。
  • FIRST_SEEN: SQL语句首次执行的时间。
  • LAST_SEEN: SQL语句最近一次执行的时间。

示例：采集锁等待信息

以下SQL语句可以从events_waits_summary_global_by_event_name表中获取全局的锁等待事件的统计信息：

SELECT
    EVENT_NAME,
    COUNT_STAR,
    SUM_TIMER_WAIT,
    AVG_TIMER_WAIT
FROM
    performance_schema.events_waits_summary_global_by_event_name
WHERE
    EVENT_NAME LIKE 'wait/lock/%'
ORDER BY
    SUM_TIMER_WAIT DESC
LIMIT 10;
  • EVENT_NAME: 事件名称，例如wait/lock/table/sql/handler。
  • COUNT_STAR: 事件发生的次数。
  • SUM_TIMER_WAIT: 事件的总等待时间（单位是皮秒）。
  • AVG_TIMER_WAIT: 事件的平均等待时间（单位是皮秒）。

示例：采集IO相关信息

SELECT
    FILE_NAME,
    COUNT_READ,
    SUM_NUMBER_OF_BYTES_READ,
    COUNT_WRITE,
    SUM_NUMBER_OF_BYTES_WRITE
FROM
    performance_schema.file_summary_by_instance
ORDER BY
    SUM_NUMBER_OF_BYTES_READ DESC
LIMIT 10;
  • FILE_NAME: 文件名.
  • COUNT_READ: 读取次数.
  • SUM_NUMBER_OF_BYTES_READ: 读取总字节数.
  • COUNT_WRITE: 写入次数.
  • SUM_NUMBER_OF_BYTES_WRITE: 写入总字节数.

采集脚本示例（Python）：

import mysql.connector
import time

def collect_performance_data(config):
    """
    Collects performance data from MySQL Performance Schema.
    """
    try:
        cnx = mysql.connector.connect(**config)
        cursor = cnx.cursor(dictionary=True)

        # SQL to collect slow query information
        slow_query_sql = """
            SELECT
                DIGEST_TEXT,
                COUNT_STAR,
                SUM_TIMER_WAIT,
                AVG_TIMER_WAIT,
                FIRST_SEEN,
                LAST_SEEN
            FROM
                performance_schema.events_statements_summary_by_digest
            WHERE
                COUNT_STAR > 0
            ORDER BY
                SUM_TIMER_WAIT DESC
            LIMIT 10;
        """

        # SQL to collect lock wait information
        lock_wait_sql = """
            SELECT
                EVENT_NAME,
                COUNT_STAR,
                SUM_TIMER_WAIT,
                AVG_TIMER_WAIT
            FROM
                performance_schema.events_waits_summary_global_by_event_name
            WHERE
                EVENT_NAME LIKE 'wait/lock/%'
            ORDER BY
                SUM_TIMER_WAIT DESC
            LIMIT 10;
        """

        # SQL to collect file IO information
        file_io_sql = """
            SELECT
                FILE_NAME,
                COUNT_READ,
                SUM_NUMBER_OF_BYTES_READ,
                COUNT_WRITE,
                SUM_NUMBER_OF_BYTES_WRITE
            FROM
                performance_schema.file_summary_by_instance
            ORDER BY
                SUM_NUMBER_OF_BYTES_READ DESC
            LIMIT 10;
        """

        # Execute queries
        cursor.execute(slow_query_sql)
        slow_queries = cursor.fetchall()

        cursor.execute(lock_wait_sql)
        lock_waits = cursor.fetchall()

        cursor.execute(file_io_sql)
        file_ios = cursor.fetchall()

        # Prepare data for storage (example: print to console)
        print("Slow Queries:")
        for row in slow_queries:
            print(row)

        print("nLock Waits:")
        for row in lock_waits:
            print(row)

        print("nFile IO:")
        for row in file_ios:
            print(row)

    except mysql.connector.Error as err:
        print(f"Error: {err}")
    finally:
        if cnx:
            cursor.close()
            cnx.close()

# Example configuration
config = {
    'user': 'your_user',
    'password': 'your_password',
    'host': 'your_host',
    'database': 'performance_schema'
}

# Run data collection periodically
while True:
    collect_performance_data(config)
    time.sleep(60)  # Collect data every 60 seconds

这个Python脚本会定期从Performance Schema中采集SQL语句执行时间、锁等待信息和IO信息，并将数据打印到控制台。你需要根据实际需求修改SQL语句和数据处理逻辑。

四、数据存储模块

数据存储模块负责存储采集到的数据。可以选择MySQL自身或其他时序数据库。

1. 使用MySQL存储:

可以在MySQL中创建一个专门用于存储监控数据的数据库和表。例如：

CREATE DATABASE IF NOT EXISTS performance_monitoring;
USE performance_monitoring;

CREATE TABLE IF NOT EXISTS slow_queries (
    id INT AUTO_INCREMENT PRIMARY KEY,
    collect_time TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    digest_text TEXT,
    count_star BIGINT,
    sum_timer_wait BIGINT,
    avg_timer_wait BIGINT,
    first_seen TIMESTAMP,
    last_seen TIMESTAMP
);

CREATE TABLE IF NOT EXISTS lock_waits (
    id INT AUTO_INCREMENT PRIMARY KEY,
    collect_time TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    event_name VARCHAR(255),
    count_star BIGINT,
    sum_timer_wait BIGINT,
    avg_timer_wait BIGINT
);

CREATE TABLE IF NOT EXISTS file_io (
    id INT AUTO_INCREMENT PRIMARY KEY,
    collect_time TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    file_name VARCHAR(255),
    count_read BIGINT,
    sum_number_of_bytes_read BIGINT,
    count_write BIGINT,
    sum_number_of_bytes_write BIGINT
);

然后，修改数据采集脚本，将采集到的数据插入到这些表中。

修改后的Python脚本示例（插入数据到MySQL）：

import mysql.connector
import time

def collect_performance_data(config):
    """
    Collects performance data from MySQL Performance Schema and stores it in a separate MySQL database.
    """
    try:
        cnx_ps = mysql.connector.connect(**config)  # Connection to Performance Schema
        cursor_ps = cnx_ps.cursor(dictionary=True)

        # Configuration for the monitoring database
        monitoring_config = {
            'user': 'monitoring_user',
            'password': 'monitoring_password',
            'host': 'monitoring_host',
            'database': 'performance_monitoring'
        }
        cnx_mon = mysql.connector.connect(**monitoring_config) # Connection to Monitoring Database
        cursor_mon = cnx_mon.cursor()

        # SQL to collect slow query information
        slow_query_sql = """
            SELECT
                DIGEST_TEXT,
                COUNT_STAR,
                SUM_TIMER_WAIT,
                AVG_TIMER_WAIT,
                FIRST_SEEN,
                LAST_SEEN
            FROM
                performance_schema.events_statements_summary_by_digest
            WHERE
                COUNT_STAR > 0
            ORDER BY
                SUM_TIMER_WAIT DESC
            LIMIT 10;
        """

        # SQL to collect lock wait information
        lock_wait_sql = """
            SELECT
                EVENT_NAME,
                COUNT_STAR,
                SUM_TIMER_WAIT,
                AVG_TIMER_WAIT
            FROM
                performance_schema.events_waits_summary_global_by_event_name
            WHERE
                EVENT_NAME LIKE 'wait/lock/%'
            ORDER BY
                SUM_TIMER_WAIT DESC
            LIMIT 10;
        """

        # SQL to collect file IO information
        file_io_sql = """
            SELECT
                FILE_NAME,
                COUNT_READ,
                SUM_NUMBER_OF_BYTES_READ,
                COUNT_WRITE,
                SUM_NUMBER_OF_BYTES_WRITE
            FROM
                performance_schema.file_summary_by_instance
            ORDER BY
                SUM_NUMBER_OF_BYTES_READ DESC
            LIMIT 10;
        """

        # Execute queries
        cursor_ps.execute(slow_query_sql)
        slow_queries = cursor_ps.fetchall()

        cursor_ps.execute(lock_wait_sql)
        lock_waits = cursor_ps.fetchall()

        cursor_ps.execute(file_io_sql)
        file_ios = cursor_ps.fetchall()

        # Insert data into monitoring tables
        for row in slow_queries:
            insert_sql = """
                INSERT INTO slow_queries (digest_text, count_star, sum_timer_wait, avg_timer_wait, first_seen, last_seen)
                VALUES (%s, %s, %s, %s, %s, %s)
            """
            values = (row['DIGEST_TEXT'], row['COUNT_STAR'], row['SUM_TIMER_WAIT'], row['AVG_TIMER_WAIT'], row['FIRST_SEEN'], row['LAST_SEEN'])
            cursor_mon.execute(insert_sql, values)

        for row in lock_waits:
            insert_sql = """
                INSERT INTO lock_waits (event_name, count_star, sum_timer_wait, avg_timer_wait)
                VALUES (%s, %s, %s, %s)
            """
            values = (row['EVENT_NAME'], row['COUNT_STAR'], row['SUM_TIMER_WAIT'], row['AVG_TIMER_WAIT'])
            cursor_mon.execute(insert_sql, values)

        for row in file_ios:
            insert_sql = """
                INSERT INTO file_io (file_name, count_read, sum_number_of_bytes_read, count_write, sum_number_of_bytes_write)
                VALUES (%s, %s, %s, %s, %s)
            """
            values = (row['FILE_NAME'], row['COUNT_READ'], row['SUM_NUMBER_OF_BYTES_READ'], row['COUNT_WRITE'], row['SUM_NUMBER_OF_BYTES_WRITE'])
            cursor_mon.execute(insert_sql, values)

        cnx_mon.commit() # Commit changes

        print("Data inserted into monitoring database successfully.")

    except mysql.connector.Error as err:
        print(f"Error: {err}")
    finally:
        if cursor_ps:
            cursor_ps.close()
        if cnx_ps:
            cnx_ps.close()
        if cursor_mon:
            cursor_mon.close()
        if cnx_mon:
            cnx_mon.close()

# Example configuration
config = {
    'user': 'your_user',
    'password': 'your_password',
    'host': 'your_host',
    'database': 'performance_schema'
}

# Run data collection periodically
while True:
    collect_performance_data(config)
    time.sleep(60)  # Collect data every 60 seconds

2. 使用时序数据库（InfluxDB）存储:

InfluxDB是专门为存储时序数据而设计的数据库，具有高写入性能和强大的时间序列分析能力。

首先，需要安装和配置InfluxDB。然后，可以使用InfluxDB的Python客户端库将数据写入到InfluxDB中。

安装 InfluxDB Python 客户端:

pip install influxdb

修改后的Python脚本示例（插入数据到InfluxDB）：

import mysql.connector
import time
from influxdb import InfluxDBClient

def collect_performance_data(config):
    """
    Collects performance data from MySQL Performance Schema and stores it in InfluxDB.
    """
    try:
        cnx = mysql.connector.connect(**config)
        cursor = cnx.cursor(dictionary=True)

        # SQL to collect slow query information
        slow_query_sql = """
            SELECT
                DIGEST_TEXT,
                COUNT_STAR,
                SUM_TIMER_WAIT,
                AVG_TIMER_WAIT,
                FIRST_SEEN,
                LAST_SEEN
            FROM
                performance_schema.events_statements_summary_by_digest
            WHERE
                COUNT_STAR > 0
            ORDER BY
                SUM_TIMER_WAIT DESC
            LIMIT 10;
        """

        # SQL to collect lock wait information
        lock_wait_sql = """
            SELECT
                EVENT_NAME,
                COUNT_STAR,
                SUM_TIMER_WAIT,
                AVG_TIMER_WAIT
            FROM
                performance_schema.events_waits_summary_global_by_event_name
            WHERE
                EVENT_NAME LIKE 'wait/lock/%'
            ORDER BY
                SUM_TIMER_WAIT DESC
            LIMIT 10;
        """

        # SQL to collect file IO information
        file_io_sql = """
            SELECT
                FILE_NAME,
                COUNT_READ,
                SUM_NUMBER_OF_BYTES_READ,
                COUNT_WRITE,
                SUM_NUMBER_OF_BYTES_WRITE
            FROM
                performance_schema.file_summary_by_instance
            ORDER BY
                SUM_NUMBER_OF_BYTES_READ DESC
            LIMIT 10;
        """

        # Execute queries
        cursor.execute(slow_query_sql)
        slow_queries = cursor.fetchall()

        cursor.execute(lock_wait_sql)
        lock_waits = cursor.fetchall()

        cursor.execute(file_io_sql)
        file_ios = cursor.fetchall()

        # Prepare data for InfluxDB
        influxdb_data = []

        for row in slow_queries:
            influxdb_data.append({
                "measurement": "slow_queries",
                "tags": {
                    "digest_text": row['DIGEST_TEXT']
                },
                "time": time.time_ns(),  # Nanoseconds
                "fields": {
                    "count_star": row['COUNT_STAR'],
                    "sum_timer_wait": row['SUM_TIMER_WAIT'],
                    "avg_timer_wait": row['AVG_TIMER_WAIT']
                }
            })

        for row in lock_waits:
            influxdb_data.append({
                "measurement": "lock_waits",
                "tags": {
                    "event_name": row['EVENT_NAME']
                },
                "time": time.time_ns(),  # Nanoseconds
                "fields": {
                    "count_star": row['COUNT_STAR'],
                    "sum_timer_wait": row['SUM_TIMER_WAIT'],
                    "avg_timer_wait": row['AVG_TIMER_WAIT']
                }
            })

        for row in file_ios:
            influxdb_data.append({
                "measurement": "file_io",
                "tags": {
                    "file_name": row['FILE_NAME']
                },
                "time": time.time_ns(),  # Nanoseconds
                "fields": {
                    "count_read": row['COUNT_READ'],
                    "sum_number_of_bytes_read": row['SUM_NUMBER_OF_BYTES_READ'],
                    "count_write": row['COUNT_WRITE'],
                    "sum_number_of_bytes_write": row['SUM_NUMBER_OF_BYTES_WRITE']
                }
            })

        # Write data to InfluxDB
        influx_client = InfluxDBClient(host='your_influxdb_host', port=8086, database='performance_monitoring')
        influx_client.write_points(influxdb_data)

        print("Data written to InfluxDB successfully.")

    except mysql.connector.Error as err:
        print(f"MySQL Error: {err}")
    except Exception as e:
        print(f"InfluxDB Error: {e}")
    finally:
        if cnx:
            cursor.close()
            cnx.close()

# Example configuration
config = {
    'user': 'your_user',
    'password': 'your_password',
    'host': 'your_host',
    'database': 'performance_schema'
}

# Run data collection periodically
while True:
    collect_performance_data(config)
    time.sleep(60)  # Collect data every 60 seconds

五、数据分析模块

数据分析模块负责分析存储的数据，计算性能指标。可以编写SQL语句或使用Python等编程语言进行分析。

示例：使用SQL分析慢查询

如果数据存储在MySQL中，可以使用以下SQL语句查询过去5分钟内执行时间超过1秒的SQL语句：

SELECT
    digest_text,
    COUNT(*) AS execution_count,
    AVG(sum_timer_wait) AS avg_execution_time
FROM
    slow_queries
WHERE
    collect_time >= NOW() - INTERVAL 5 MINUTE
    AND sum_timer_wait > 1000000000000  -- 1 second in picoseconds
GROUP BY
    digest_text
ORDER BY
    avg_execution_time DESC;

示例：使用InfluxDB分析锁等待

如果数据存储在InfluxDB中，可以使用以下InfluxQL查询过去5分钟内锁等待时间超过100毫秒的事件：

SELECT
    mean(sum_timer_wait)
FROM
    lock_waits
WHERE
    time >= now() - 5m
    AND sum_timer_wait > 100000000000  -- 100 milliseconds in picoseconds
GROUP BY
    event_name;

六、预警模块

预警模块负责根据预设的阈值，触发预警。可以编写Python脚本或其他工具来实现预警功能。

示例：Python预警脚本

import mysql.connector
import time
import smtplib
from email.mime.text import MIMEText

def check_slow_queries(config, threshold):
    """
    Checks for slow queries in the last 5 minutes and sends an alert if the threshold is exceeded.
    """
    try:
        cnx = mysql.connector.connect(**config)
        cursor = cnx.cursor()

        sql = """
            SELECT
                digest_text,
                COUNT(*) AS execution_count,
                AVG(sum_timer_wait) AS avg_execution_time
            FROM
                slow_queries
            WHERE
                collect_time >= NOW() - INTERVAL 5 MINUTE
                AND sum_timer_wait > %s  -- Threshold in picoseconds
            GROUP BY
                digest_text
            ORDER BY
                avg_execution_time DESC;
        """
        cursor.execute(sql, (threshold,))
        slow_queries = cursor.fetchall()

        if slow_queries:
            # Send alert email
            send_alert_email(slow_queries)

    except mysql.connector.Error as err:
        print(f"Error: {err}")
    finally:
        if cnx:
            cursor.close()
            cnx.close()

def send_alert_email(slow_queries):
    """
    Sends an alert email with information about slow queries.
    """
    sender_email = "[email protected]"
    receiver_email = "[email protected]"
    password = "your_email_password"

    message = MIMEText(f"Slow queries detected:n{slow_queries}")
    message['Subject'] = "MySQL Slow Query Alert"
    message['From'] = sender_email
    message['To'] = receiver_email

    try:
        with smtplib.SMTP_SSL('smtp.gmail.com', 465) as server:
            server.login(sender_email, password)
            server.sendmail(sender_email, receiver_email, message.as_string())
        print("Alert email sent successfully.")
    except Exception as e:
        print(f"Error sending email: {e}")

# Example configuration
config = {
    'user': 'monitoring_user',
    'password': 'monitoring_password',
    'host': 'monitoring_host',
    'database': 'performance_monitoring'
}

# Threshold for slow query (1 second in picoseconds)
threshold = 1000000000000

# Run check periodically
while True:
    check_slow_queries(config, threshold)
    time.sleep(60)  # Check every 60 seconds

这个脚本会定期检查过去5分钟内是否存在执行时间超过1秒的SQL语句，如果存在，则发送警报邮件。

七、展示模块

展示模块负责展示监控数据和预警信息。可以使用Grafana等可视化工具，也可以自定义Web界面。

Grafana:

Grafana是一个流行的开源数据可视化工具，可以连接到MySQL或InfluxDB，并创建各种图表和仪表盘来展示监控数据。

自定义Web界面:

可以使用Python的Flask或Django框架，搭建一个Web界面来展示监控数据和预警信息。

八、Performance Schema 配置优化

默认情况下，Performance Schema的所有instrumentation都是禁用的。我们需要根据监控需求，启用特定的instrumentation。

1. 启用Instrumentation:

可以使用以下SQL语句启用instrumentation：

UPDATE performance_schema.setup_instruments SET ENABLED = 'YES' WHERE NAME LIKE '%statement/%';
UPDATE performance_schema.setup_instruments SET ENABLED = 'YES' WHERE NAME LIKE '%wait/lock/%';
UPDATE performance_schema.setup_instruments SET ENABLED = 'YES' WHERE NAME LIKE '%file/%';

2. 调整Consumer配置:

可以使用以下SQL语句调整consumer配置：

UPDATE performance_schema.setup_consumers SET ENABLED = 'YES' WHERE NAME LIKE '%statement/%';
UPDATE performance_schema.setup_consumers SET ENABLED = 'YES' WHERE NAME LIKE '%wait/lock/%';
UPDATE performance_schema.setup_consumers SET ENABLED = 'YES' WHERE NAME LIKE '%file/%';

3. 调整History Size:

Performance Schema的表都有一个history size，用于控制存储多少历史数据。可以根据需要调整history size。例如：

SET GLOBAL events_statements_history_long_size = 10000;

九、总结：构建实时监控，保障MySQL健康

通过以上步骤，我们就能够构建一个基于MySQL Performance Schema的实时性能监控与预警系统。这个系统可以帮助我们深入了解MySQL服务器的运行状态，及时发现和解决性能问题，保障MySQL服务器的稳定运行。

十、一些优化建议

  • 定期清理监控数据： 为了避免监控数据占用过多的存储空间，需要定期清理不再需要的数据。
  • 设置合理的预警阈值： 预警阈值设置过高可能会导致错过一些重要的告警，设置过低可能会导致误报。需要根据实际情况设置合理的预警阈值。
  • 优化SQL语句： 确保用于采集和分析数据的SQL语句是高效的，避免对MySQL服务器造成额外的负担。
  • 监控Performance Schema的开销： Performance Schema虽然设计为低开销，但过度使用仍然可能对MySQL服务器的性能产生影响。需要定期监控Performance Schema的开销，并根据需要调整配置。

希望今天的分享能帮助大家更好地利用MySQL Performance Schema进行性能监控和调优。 谢谢大家。

发表回复

您的邮箱地址不会被公开。 必填项已用 * 标注