1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
//! x2APIC Emulation
//!
//! ## Background
//! ### Advanced Programmable Interrupt Controller
//! In interrupt handling, the Advanced Programmable Interrupt Controller (APIC) represents
//! a significantly improved alternative to the conventional Programmable Interrupt Controller (PIC).
//! Originally, the 8235 PIC chip was responsible for interrupt processing, but due to speed and functionality considerations,
//! interrupt handling routine was refined and integrated into the CPU in the cases of the local core interrupts.
//!
//! APIC is implemented as separate Local APIC and IOAPIC components.
//! The former is embedded within each processor, while the latter is located in the system bus.
//! APIC's primary objective is to process interrupts on processors and forward them to designated cores,
//! as well as facilitate Inter-Processor Interrupt (IPI) communication in multicore environments.
//!
//! Over time, Intel has enhanced the APIC to newer versions such as xAPIC and x2APIC.
//! One of the main differences between APIC and xAPIC is their functionality
//! (Increased number of interrupt lines and flexibility in handling different types of interrupts),
//! while x2APIC offers the ability to modify performance behavior through the use of MSR registers.
//!
//! #### LAPIC
//! Each core has a dedicated Local APIC assigned to it, which can generate interrupts and trigger Inter-Processor Interrupts (IPIs)
//! as well as handle up to 0-31 interrupt processing and 32-225 designated interrupt processing.
//! Local APIC, which is directly connected to the processor's cores, performs interrupt-related tasks
//! such as handling I/O devices, APIC Timer, IPI, Performance Monitoring Counter, and Thermal Sensor.
//! LAPIC supports High Precision Event Timer (HPET), which is a high-precision timer that assists each core in obtaining
//! accurate current time values without competing for one timer. If APIC is enabled and HPET is supported, the 8253 PIC is disabled.
//!
//! The definition of how each interrupt behaves can be configured via the Local Vector Table (LVT).
//! LVT is controlled through memory access via MMIO for APIC, xAPIC, and similar devices, and via MSR registers for x2APIC.
//! The Local Vector Table determines which hardware interrupt is forwarded to the core's interrupt pin.
//!
//! #### I/O APIC
//! The I/O APIC is an Interrupt Controller that is responsible for delivering external device interrupts to the appropriate cores.
//! Interrupts delivered through the I/O APIC are forwarded to the Local APIC based on the LVT configuration and ultimately delivered to the core.
//! The I/O APIC is connected to the system bus and is responsible for delivering interrupts generated by hardware or I/O devices to the core.
//!
//! The I/O APIC has a Redirection Table that maps incoming interrupts from each device to a designated interrupt number.
//! Unlike the 8253 interrupt controller, which supports only 16 external interrupts, the APIC supports up to 224 additional interrupts,
//! excluding those assigned to hardware, through redirection.
//!
//! ### APIC Timer - TSC deadline mode
//! APIC Timer modes are separated into three different modes for handling timer interrupts.
//!
//! #### Periodic Mode and One-shot Mode
//! The Periodic Mode is a mode in which software (usually the operating system) requests the
//! APIC to generate periodic timer interrupts by setting a specific time interval.
//! For example, in the Periodic Mode, software requests the APIC to insert a timer interrupt every 1ms.
//!
//! The One-shot Mode is similar to the Periodic Mode but requests the timer interrupt to occur only once.
//! Software requests the APIC to generate a timer interrupt after a certain time interval.
//! For example, software requests the APIC to insert a timer interrupt 1ms later.
//!
//! #### TSC-Deadline Mode
//! The TSC-Deadline Mode has a slightly different characteristic from the other two timer modes.
//! Rather than sending a specific timer interval, this mode requests a timer interrupt when the value of the
//! Time Stamp Counter (TSC) of the core exceeds a specific value.
//! This method uses the TSC timer, which is more accurate than the other two methods that use the CPU frequency,
//! and has a feature that is easy to avoid race conditions.
//! For example, if the current TSC count is 100000, software requests the APIC to generate a timer interrupt when it becomes 100050,
//! which is 100000 + α. Like one-shot mode, software should reprogram the next timer manually to re-trigger the next timer interrupt.
//!
//! Note that, the TSC deadline setting and initiate interrupts are separated.
//! Initiate x2APIC timer interrupts are done by setting Model Specific Register (MSR register) with 0x832 to 0x10 (TSC-Deadline Mode).
//! However, setting a deadline for Local APIC's TSC Deadline Mode is done by setting MSR with 0x6e0 to a timestamp for the next deadline.
//!
//! ## Tasks
//! In this project, you are requested to implement timer interrupt virtualization for guest operating system.
//! You need to implement timer interrupt virtualization for the guest by ensuring that
//! a timer interrupt occurs every tick to schedule threads in the guest operating system.
//!
//! As the guest is unable to use the APIC timer in the host,
//! the host must handle the initialization of the timer and setting of the deadline from the guest.
//! If the TSC value exceeds the guest deadline, a virtual interrupt should be injected into the guest
//! by a separate thread that runs in the hypervisor.
//! This thread is spawned when the guest requests to set the timer as TSC deadline mode and
//! tries to write the timer mode and interrupt vector into the MSR 0x832 (the MSR write is trapped to guest).
//!
//! After the thread is spawned,
//! a deadline setting request via 0x6e0 MSR can be sent to the thread to inject interrupts when the TSC value is more than the set deadline.
//! To share the deadline value between the created thread and the handler for 0x6e0 MSR, the [`channel`] API provided by KeV is used.
//! Injection of the interrupt into the VM should only be done when the VM is not running, as the injected interrupt is handled when VmEntry occurs.
//! To inject the timer interrupt into the running vCPU, the VMM must 1) [`kick`] the vCPU, 2) [`inject`] the interrupt,
//! and then 3) [`resume`] the vCPU to execute the timer interrupt in the guest.
//!
//! [`channel`]: keos::thread::channel::channel
//! [`kick`]: kev::vm::VmOps::kick_vcpu
//! [`inject`]: kev::vcpu::VCpuOps::inject_interrupt
//! [`resume`]: kev::vm::VmOps::resume_vcpu

use alloc::sync::Arc;
use core::arch::x86_64::_rdtsc;
use keos::{
    spin_lock::SpinLock,
    thread::{
        channel::{channel, Sender},
        ThreadBuilder,
    },
};
use kev::{vcpu::GenericVCpuState, vm::Gpa, Probe, VmError};
use project2::vmexit::msr;

/// X2Apic internal state
pub struct X2ApicInner {
    apic_base_0x1b: u64,
    tx: Option<Sender<u64>>,
}

/// X2Apic
pub struct X2Apic {
    inner: Arc<SpinLock<X2ApicInner>>,
}

const APIC_BASE: u64 = 0xfee0_0800;

impl X2Apic {
    pub fn attach(ctl: &mut msr::Controller) {
        let inner = Arc::new(SpinLock::new(X2ApicInner {
            apic_base_0x1b: APIC_BASE,
            tx: None,
        }));

        // APIC_BASE MSR.
        assert!(ctl.insert(
            0x1B,
            X2Apic {
                inner: inner.clone()
            }
        ));
        // TP.
        assert!(ctl.insert(
            0x808,
            X2Apic {
                inner: inner.clone()
            }
        ));
        // eoi
        assert!(ctl.insert(
            0x80b,
            X2Apic {
                inner: inner.clone()
            }
        ));
        // icr
        assert!(ctl.insert(
            0x830,
            X2Apic {
                inner: inner.clone()
            }
        ));
        // timer
        assert!(ctl.insert(
            0x832,
            X2Apic {
                inner: inner.clone()
            }
        ));
        // Susprious interrupt vector.
        assert!(ctl.insert(
            0x80F,
            X2Apic {
                inner: inner.clone()
            }
        ));
        // lint0
        assert!(ctl.insert(
            0x835,
            X2Apic {
                inner: inner.clone()
            }
        ));
        // lint1
        assert!(ctl.insert(
            0x836,
            X2Apic {
                inner: inner.clone()
            }
        ));

        // tsc_deadline
        assert!(ctl.insert(
            0x6e0,
            X2Apic {
                inner: inner.clone()
            }
        ));
    }
}

impl msr::Msr for X2Apic {
    fn rdmsr(
        &self,
        index: u32,
        _p: &dyn Probe,
        _generic_vcpu_state: &mut GenericVCpuState,
    ) -> Result<u64, VmError> {
        let inner = self.inner.lock();
        match index {
            0x1b => Ok(inner.apic_base_0x1b),
            0x808 | 0x80b | 0x830 | 0x80f | 0x835 | 0x836 | 0x832 | 0x6e0 => Ok(0),
            _ => unreachable!(),
        }
    }

    fn wrmsr(
        &mut self,
        index: u32,
        value: u64,
        p: &dyn Probe,
        generic_vcpu_state: &mut GenericVCpuState,
    ) -> Result<(), VmError> {
        let mut inner = self.inner.lock();
        match index {
            0x1b => inner.apic_base_0x1b = value,
            // ICR
            0x830 => {
                let icr = ICR::from_bits_truncate(value as u32);
                let (dst, _ipi) = ((value >> 32) as u32, value as u8);
                match icr.mode() {
                    ICRMode::Init => (),
                    ICRMode::StartUp => {
                        let entry = unsafe {
                            (*p.gpa2hva(&generic_vcpu_state.vmcs, Gpa::new(0x467).unwrap())
                                .unwrap()
                                .as_mut::<u16>()
                                .unwrap()
                                << 4)
                                | (*p
                                    .gpa2hva(&generic_vcpu_state.vmcs, Gpa::new(0x469).unwrap())
                                    .unwrap()
                                    .as_mut::<u16>()
                                    .unwrap()
                                    & 0xf)
                        };
                        let _ = generic_vcpu_state
                            .vm
                            .upgrade()
                            .unwrap()
                            .start_vcpu(dst as usize, entry);
                    }
                    _ => panic!("Unsupported"),
                }
            }
            0x832 => {
                let mode = (value >> 17) & 0b11;
                let int = value as u8;
                assert_eq!(mode, 0b10, "Only tsc deadline mode is supported.");
                let (tx, rx) = channel(1);
                todo!();

                ThreadBuilder::new("timer_intr").spawn(move || {
                    // Hint:
                    //    - Receive the deadline from the rx.
                    //    - Wait until time stamp exceeds the deadline.
                    //    - You can get time stamp count with _rdtsc().
                    //    - Kick vcpu and inject the interrupt #int to the vcpu.
                    //    - Resume vcpu.
                    todo!()
                });
                inner.tx = Some(tx);
            }
            0x6e0 => {
                todo!()
            }
            0x808 | 0x80b | 0x80f | 0x835 | 0x836 => (),
            _ => unreachable!(),
        }
        Ok(())
    }
}

bitflags::bitflags! {
    struct ICR: u32 {
        const DEST_1 = 1 << 19;
        const DEST_0 = 1 << 18;
        const LEVEL = 1 << 15;
        const ASSERT = 1 << 14;
        const PHYSICAL = 1 << 11;
        const MODE_2 = 1 << 10;
        const MODE_1 = 1 << 9;
        const MODE_0 = 1 << 8;
    }
}

impl ICR {
    fn mode(&self) -> ICRMode {
        match (self.bits() >> 8) & 7 {
            0b000 => ICRMode::Fixed,
            0b010 => ICRMode::Smi,
            0b100 => ICRMode::Nmi,
            0b101 => ICRMode::Init,
            0b110 => ICRMode::StartUp,
            _ => ICRMode::Reserved,
        }
    }
}

#[derive(Debug)]
enum ICRMode {
    Fixed,
    Smi,
    Nmi,
    Init,
    StartUp,
    Reserved,
}